AI token optimization: 5 ways to keep your LLM from burning through your IT budget

Summary
AI inference is being treated like a software licence and ends up surprising people with its actual impact on their IT budgets. You do not buy a licence, you buy tokens, and the same feature can cost ten times more this month than last because a prompt changed. The most overlooked is that output tokens cost five times what input tokens cost, so a verbose response is a more expensive mistake than a long prompt.
So here are five strategies to identify the token consumption multipliers rand control costs.
Most companies discovered AI spend the way they discovered cloud spend: through an invoice nobody could explain. The FinOps Foundation's 2026 research records the shift bluntly, with 98% of practitioners now managing AI spend against 31% two years earlier. Almost nobody was watching this two years ago, and almost everybody is now.
AI token optimization means reducing what you pay for the text going into and out of a model, without reducing what the model does for you. Let's see how you can do it.
What AI token optimization actually means
In any AI usage, you pay for three things: the text you send, the text the model sends back, and in some cases a fee for storing a reusable copy of what you sent. Those three are priced differently, and that is the gap we will be targeting for our AI token optimization.
Let's take a real-world case of Claude by Anthropic. On the published Claude API rates, Claude Opus 5 costs $5 per million input tokens and $25 per million output tokens. Sonnet 5 is $2 and $10. Haiku 4.5 is $1 and $5. In every case, output costs five times what input costs.
Most teams optimize the prompt, because the prompt is the part they wrote and can see. But the output is where the cost is actually consumed. A model instructed to explain its reasoning at length in every reply is a standing cost decision that nobody made deliberately.
Two more things worth knowing before the list. These figures are one provider's published rates, used here because they are public and specific; the pricing structure is broadly similar across major providers, but check your own. And rates move: the same documentation shows Sonnet 5 priced at $2 per million input tokens through 31 August 2026 and $3 from 1 September. Any business case built on today's numbers has an expiry date on it and has to be mindful of where their cost goes.
Five proven strategies for AI token optimization
Ordered by how much they typically save. "Proven" here means the provider publishes the multiplier, not that an independent study validated a saving. What you actually save depends on the shape of your traffic, which is the point of the measurement section further down.
1. Cache the part of the prompt that never changes
Most production prompts are mostly boilerplate: instructions, tool definitions, a policy document, examples. That prefix is identical on every call and you pay full price for it every time.
Caching charges you differently. According to the Claude API documentation, a cache read costs only 0.1 times than reading a user prompt. However, writing the cache itself costs 1.25 times as the user input.
Every time you trigger the cache, you need to be careful that your prefix matches the cache completely. If you inject new data such as a timestamp or user ID, then the cache breaks and you pay the write cost again.
There is also a lower limit for caching. Prompts below a minimum length are not cached at all and no error is returned, and the minimum varies by model, from 512 tokens on Opus 5 to 4,096 on Haiku 4.5. A team can believe caching is on for months while every call silently misses.
2. Send anything that can wait through the batch endpoint
If a workload does not need an answer in the next second, it should not be paying real-time rates. The batch documentation puts the discount at 50% on all usage, input and output alike, with most batches finishing inside an hour against a 24-hour window.
The discount stacks with caching, which the documentation states directly. That combination is the cheapest way to run bulk work, with one caveat also documented: because batch requests run asynchronously, cache hits are best-effort, and observed hit rates range from 30% to 98% depending on traffic patterns. Wide range, so measure yours rather than assuming the top of it.
Good candidates: overnight enrichment, document classification, backfills, report generation, anything triggered by a schedule instead of a person.
3. Stop sending every task to your most expensive model
The spread between the cheapest and most expensive model in a single family is five times on input and five times on output. Plenty of production traffic is classification, extraction, routing, and summarisation, none of which needs the strongest model available.
This is the strategy with the most organisational friction, because it requires someone to decide per task rather than picking one model for everything. It is also the one most likely to degrade quality without anyone noticing, so it belongs behind an evaluation set rather than a preference. Move a task down a tier, measure whether the output still passes, keep it there if it does.
4. Budget your output, not only your prompt
Given that output costs five times input, the highest-return prompt change is usually an instruction about length. Ask for the answer without the preamble. Use structured output where you need fields rather than prose, because a schema constrains length by construction. Set a maximum on response length as a ceiling, not as a target.
Reasoning depth is the version of this that costs most and gets watched least. Where a provider exposes a control over how much the model thinks before answering, that control is a spend dial. Left at maximum across all traffic, it charges premium output rates for deliberation on questions that did not need it.
5. Bound the context instead of letting it grow
In long agent runs the context grows every turn, and you pay for the whole thing on every call. Two documented mechanisms address this. Compaction watches input tokens against a threshold, writes a summary, and then drops everything before that summary on later requests. Context editing clears old tool results, which is aimed at agents whose context fills with large file reads and API responses they could fetch again if needed.
One correction worth carrying to your architects, from the context windows documentation: a cached prefix still occupies the context window. Caching changes what those tokens cost, not whether they count. Teams routinely conflate the two and are surprised when a heavily cached prompt still hits a context limit.
The tradeoff no pricing page mentions
Strategies one and five fight each other, and this is documented rather than theoretical. Clearing old tool results invalidates cached prompt prefixes, per the context editing documentation. Anything that edits the cached portion of a conversation destroys the cache entry that portion belonged to.
So an agent platform that enables both, on the reasonable assumption that two savings add up, can end up paying cache write premiums repeatedly while getting few reads. The bill goes down less than either change predicted, and the team cannot tell which lever failed because both moved at once.
The practical rule is unglamorous: change one lever at a time, and read the usage figures after each one before touching the next.
Which is why someone has to own this decision rather than leave it to whoever gets there first. The two strategies belong to different people. Caching is a prompt-structure decision owned by whoever writes the prompts. Context clearing is a runtime decision owned by whoever builds the agent. Neither can see the other's effect on the bill.
What to measure before you change anything
Every one of these strategies is unverifiable without per-call token accounting, and most teams do not have it. The provider returns the numbers; the work is capturing them.
At minimum, record for every call: tokens read from cache, tokens written to cache, uncached input tokens, and output tokens. The Claude API reports these as cache_read_input_tokens, cache_creation_input_tokens, input_tokens, and output_tokens, and per the caching documentation total input is the sum of the first three rather than the input_tokens field alone. Reading that field as the whole input is a common accounting error, and it understates cost.
Three derived figures are what you actually manage:
Cache hit rate, per feature rather than in aggregate. A sudden drop in hits and a spike in writes means someone changed a prompt prefix, usually without knowing it was cached.
Output-to-input token ratio, per feature. Rising ratio means responses are getting longer without anyone deciding they should.
Cost per completed task, not cost per call. Cost per call falls when you split one call into three, which is not a saving.
A team without these numbers is not optimizing tokens. It is changing settings and hoping.
Where token optimization stops being worth it
This work has a floor, and pretending otherwise wastes engineering time on a rounding error.
If your monthly inference spend is small enough that halving it changes nothing you would notice, the correct answer is to instrument usage and revisit when volume grows. Caching pays back on repeated prefixes at volume; on low-traffic features the write premium can exceed the read savings. Batching pays back on scheduled work; on interactive features it is not available at any price, because the latency is the product.
What this costs: engineering time on prompt restructuring and telemetry plumbing, measured in weeks rather than quarters, and an evaluation set for any model you move down a tier, which is the part teams skip and later regret. It also costs some architectural flexibility, because a cached prefix is a prompt you have agreed to stop editing casually.
Worth doing when inference is a visible line item and your traffic has repeated structure. Not worth doing when it is neither, in which case measure and wait.
For CXOs, the question to raise next: ask what your output-to-input token ratio is on your highest-volume AI feature. If nobody can answer, the AI budget is being set by prompt-writing habits rather than by anyone's decision.
FAQ
What is AI token optimization?
AI token optimization means reducing what you pay for the text going into and out of a language model, without reducing what it does for you. The main levers are caching repeated prompt content, batching work that can wait, matching tasks to cheaper models, limiting response length, and stopping conversation context from growing without bound.