How much an AI feature costs to run: a worked example

What does it cost to run an AI feature in production? Most teams find out after launch, which is the expensive way to learn that output tokens are billed at five times input.

Here’s the arithmetic, worked through with real published prices, plus the three costs that surprise people.

Tokens are the billing unit

You’re charged per token, not per request. A token averages roughly four characters of English, so a thousand tokens is about 750 words.

Input and output are priced separately, and the gap is large. Generating text costs more than reading it, because generation runs the model once per token produced.

That asymmetry comes straight from the architecture. In a transformer, described in Attention Is All You Need, reading input can be processed in parallel while output has to be produced one token at a time.

Model	Input / 1M	Output / 1M
Grok 4.6	$2	$6
GPT-5.6 Sol	$5	$30
Self-hosted open weights	Hardware and operations only

That five-fold output gap is the single most important number in any cost model, and the published figures put input roughly 60% apart too.

AI feature cost, worked through

Take a support assistant. Each conversation sends about 3,000 input tokens of context and history, and generates about 500 output tokens in reply.

At $2 and $6 per million, that’s 0.6 cents input plus 0.3 cents output, so roughly 0.9 cents per conversation. Ten thousand conversations a day is about $90 daily, or $2,700 a month.

Run the same volume at $5 and $30 and you get 1.5 cents plus 1.5 cents, so 3 cents per conversation. That’s $300 a day, or $9,000 a month.

Same product, same traffic, same measured capability. The difference between model choices is $6,300 a month.

Surprise one: reasoning tokens

Reasoning models generate intermediate working before answering, and that working is billed as output even though you never see it.

A model that thinks for 2,000 tokens before writing a 200-token answer has charged you for eleven times the visible result. Our piece on what thinking tokens buy covers when that’s justified.

The fix is routing rather than avoidance. Send extraction, classification and summarisation to a fast model, reserve reasoning for genuinely multi-step work, and measure the split.

The gains are real where the task suits it. DeepSeek’s R1 work reported strong results on mathematics and code, which share the property that answers can be checked mechanically. Most product traffic doesn’t look like that.

Surprise two: context is resent every time

Models have no memory between requests. A chat that appears to remember your conversation is resending the entire history with every message.

So a long conversation costs quadratically rather than linearly. Turn twenty pays for turns one through nineteen again, which is why chat features get expensive at exactly the point users find them useful.

Position matters as well as volume. Stanford’s Lost in the Middle found accuracy “significantly degrades when models must access relevant information in the middle of long contexts”, so padding context costs money and accuracy at once.

Prompt caching is the main defence. Most providers let you cache a stable prefix so repeated queries against the same context pay full price once, which our piece on context windows explains in more detail.

Surprise three: retries and failures

Every retry is billed. A request that times out, returns malformed JSON, or fails validation costs the same as one that worked.

In agent systems this compounds badly, because a loop that fails at step eight has paid for eight steps and produced nothing. That failure mode is the subject of our piece on why agents stall in production.

Budget for a retry rate rather than assuming a clean path, and put a hard spend cap on any loop that can call itself.

Four ways to cut the bill

Route by task first, because it’s the largest saving available and costs nothing but engineering. Most production traffic doesn’t need a frontier model.

Cache aggressively, since a stable system prompt resent thousands of times is pure waste. Trim context rather than padding it, because more context isn’t free accuracy and you pay per token either way.

Constrain output length explicitly. Asking for three bullets instead of leaving length open is a direct cut to the expensive half of the bill.

When self-hosting starts making sense

Self-hosting converts a per-token cost into a fixed hardware cost, which is a terrible deal at low volume and a good one at high volume.

The crossover is roughly where your monthly token spend exceeds what suitable hardware costs to rent. Meta’s 30B open-weight release under Apache 2.0 is sized to run on consumer hardware, which moves that line closer than it used to be.

Do not underestimate the operational burden, though. Our comparison of open weights against a closed API covers what you take on alongside the savings.

Why prices keep moving

Any cost model you build has a short shelf life, and that’s mostly good news.

Epoch AI reports pre-training compute efficiency improving roughly 3x per year, and serving efficiency improves alongside it. Architecture helps too, since mixture-of-experts models activate a fraction of their parameters per token.

The counter-pressure is that capability keeps arriving in more expensive forms. Reasoning models cost more per request than the models they replace, so the cheapest capability gets cheaper while the newest stays dear.

Checking AI feature cost before you launch

Instrument token usage per request from day one, split by input, output and cached, because you can’t optimise a bill you can’t see.

Then model your worst case rather than your average. Cost scales with success, and the scenario that breaks budgets is the one where the feature works and everybody uses it.

One structural point worth holding onto. Unlike ordinary software, every use of an AI feature has a marginal cost, which is why bundling it into a flat subscription makes your heaviest users your least profitable ones.

That’s the same pressure pushing vendors toward metered pricing, and free open-weight releases toward becoming the default floor for anyone able to run them.