The advertised price per token is the least useful number for predicting your LLM bill. The real cost is driven by how much context you send on every call, how often you retry, how many retrieval and evaluation calls sit behind each user request, and how much of that traffic you could have avoided entirely. A model that looks cheap per token becomes expensive at scale because the token count per useful answer is far higher than the headline figure suggests. The pricing page describes the meter. Your architecture decides how fast it spins.
Why per-token pricing misleads
Per-token pricing invites you to think about the cost of one call in isolation. In production there is no such thing as one call. A single user action often triggers a retrieval step, one or more model calls, perhaps a second model checking the first, and a retry when something times out. The user sees one answer. Your bill sees five calls, each carrying a full context window. When people are shocked by their first real invoice, this is almost always why. They budgeted for the visible call and paid for the invisible ones.
Where the money actually goes
The genuine cost drivers are rarely the ones teams watch.
- Context length. This is the big one. Every token you put in the prompt is billed on every call, and it is easy to grow the context without noticing. A long system prompt, a full conversation history, several retrieved documents: send all of that on every turn and your input tokens dwarf your output tokens. Most bills are context, not answers.
- Retries. Timeouts, rate limits and malformed responses all trigger retries, and every retry is a full paid call. A modest failure rate multiplied across high volume is a real line item, not rounding.
- Retrieval calls. If you fetch context before each generation, you are paying for embeddings and search too. At scale these are not free, and a chatty retrieval layer adds up quietly.
- Evaluation. Using a model to grade another model, or running quality checks on outputs, is good practice and it doubles your call volume on the traffic it covers. Worth it, but budget for it deliberately rather than being surprised.
Most teams optimise the model choice and ignore the context length. It is usually backwards. The cheapest saving is the token you never send.
The question people skip: does this need an LLM at all
The most effective cost control is refusing to use a language model where you do not need one. LLMs are extraordinary at ambiguity and language. They are a wasteful choice for tasks that are actually deterministic. Before an LLM earns a place in a hot path, I want to know why a cheaper option will not do.
- A rule beats a model when the logic is fixed. Routing by keyword, validating a format, checking a threshold: write the rule. It is faster, free, and it cannot hallucinate.
- A smaller model beats a large one for classification, extraction and routing. Reserve the expensive model for the genuinely hard reasoning and let a small, cheap model handle the high-volume simple work. Matching model size to task difficulty is often the single largest saving available.
- A traditional approach beats both for search, ranking and structured lookups that were solved problems long before LLMs arrived.
None of this is anti-AI. It is spending your model budget where it creates value instead of burning it on work a five-line function would do better.
Caching: the saving most teams leave on the table
A large share of production traffic is repetitive. The same questions, the same documents, the same fixed instructions on every call. Caching turns that repetition into savings.
- Response caching. If an identical request has been answered recently and the answer is stable, serve the stored answer. No call, no cost, faster response.
- Prompt caching. When a large, unchanging block of context sits at the front of every call, caching it means you stop paying full price to resend the same tokens repeatedly. For long shared system prompts this is a meaningful reduction with almost no downside.
Caching is unglamorous and it is often the highest-return work available on an LLM bill. It rarely gets prioritised because it does not add a feature. It just quietly removes cost, which is exactly why it is worth doing early.
Budgeting for scale
If you are planning for real volume, do not budget from the headline token price. Budget from the full cost of a completed task, then multiply by realistic traffic. A few habits keep the numbers honest:
- Measure cost per task, not per token. Include every call behind one user action: retrieval, generation, evaluation and the retries. That figure is your real unit economics.
- Set budgets and alerts. A runaway loop or a spike in retries should page someone before it appears on an invoice. Treat cost as a metric you monitor, not a number you discover monthly.
- Model your worst case. Costs scale with the longest contexts and the highest retry rates, not the averages you demoed. Budget for the tail, because production lives there.
The teams that stay in control are the ones that treat cost as an engineering constraint from day one, the same way they treat latency. It is far cheaper to design for it than to discover it.
Getting these decisions right, which tasks need a model, where to cache, how to size things, is core to our AI development work. If a growing LLM bill is on your mind, book a free consultation and we will look at where the money is actually going.