# Prompt caching: A practical guide to reducing AI token usage and costs **FinOps for AI** ![Elizebeth Joseph](https://cdn.manageengine.com/sites/meweb/images/cloudspend/elizebeth-joseph.png) **Elizebeth Joseph** Product Specialist at CloudSpend Last updated: July 2026 6 min read Every API call to an AI model costs tokens. What most teams don't realize is how many of those tokens are identical across calls: The same system prompt sent on every request. The same reference document resent each time a user asks a question. The same conversation history rebuilt from scratch on every turn. At low volume, this doesn't register. At scale, it shows up on the bill. A customer support bot sending a 2,000-token system prompt on every call is paying for those tokens hundreds of thousands of times a month. Prompt caching stops that. You pay for those tokens once at full price and at a steep discount every time after. ## What is prompt caching? When you call an AI model, you send a prompt. Usually it's a system instruction, some context, and the user's input. The model processes all of it and generates a response. You pay for every token processed. Prompt caching lets you mark certain parts of that prompt as cacheable. The first time the model sees it, it processes and stores it. On subsequent calls where the cached section matches, the model reads from the cache instead of reprocessing. You still pay for cached tokens but at a much lower rate. On AWS Bedrock, cached input tokens cost around 90% less than standard input tokens for supported models. On Azure OpenAI, caching kicks in automatically for prompts over a certain length, with cached tokens billed at around 50% of the standard rate. There is one constraint worth knowing: The cached section has to sit at the beginning of the prompt and stay consistent across calls. Dynamic content like the user's question, real-time data, and per-request variables goes after the cached section and gets processed at the normal rate. ## The token math The savings depend on how much of your prompt is static versus dynamic. Take a customer support bot sending a 2,000-token system prompt on every call. Instructions, tone guidelines, product information, escalation rules. The user's message adds another 50–100 tokens on top. Without caching, every call processes roughly 2,100 input tokens. At Claude Sonnet's rate of $3 per million input tokens, 500,000 calls a month works out to around $3,150 in input token costs. With caching, the 2,000-token system prompt gets stored after the first call. Each subsequent call processes 100 tokens at the standard rate, plus 2,000 cached tokens at roughly $0.30 per million. The same 500,000 calls now costs around $345. **That's around $2,800 a month saved—an 89% reduction in input token costs, from one change with no impact on what the bot actually produces.** ## Where teams are paying for the same tokens repeatedly ### Large system prompts sent on every call System prompts that include detailed instructions, persona guidelines, product context, or safety rules can easily run to 1,000–5,000 tokens. If this content is static across calls, every token in it is a caching opportunity being missed. ### Reference documents queried multiple times A user uploads a 10-page contract and asks five questions about it. Without caching, the document gets sent to the model five times. With caching, it goes once, and the subsequent queries run against the cached version. The same logic applies to a product catalog, policy documents, or any large reference material that gets queried repeatedly in a session. ### Multi-turn conversation history In a multi-turn conversation, each new message typically resends the full history so the model has context. By turn 10, you're resending nine previous turns on every call. Caching the earlier turns keeps the per-call token count from growing out of control. ### RAG pipelines with repeated chunks Some retrieval pipelines consistently pull the same chunks for similar queries. This is common in domain-specific applications like legal tools or internal knowledge bases. Those chunks are worth caching. Instead of retrieving and resending them on every call, cache them once and query against the stored version. ## 4 caching strategies and when to use them Not every workload benefits from caching in the same way. Here are the four main approaches and where each one makes sense. ### 1. System prompt caching If your system prompt is static or changes infrequently, cache it. Every call after the first processes those tokens at the cached rate. Works well for chatbots, internal tools, coding assistants, and any application with a consistent instruction set. This is usually where teams start, and for most it delivers the biggest immediate saving. ### 2. Document caching Best for applications where users interact with a specific document over multiple turns. A legal tool where a lawyer asks several questions about a contract. A research assistant referencing a paper across a session. Cache the document at the start of the session and all subsequent queries use the cached version rather than resending the full document each time. ### 3. Conversation history caching As a conversation grows, the cost per turn grows with it. Caching earlier turns keeps the per-call token count manageable. Particularly useful for long-running agent workflows or customer support conversations that span many exchanges. ### 4. RAG context caching If your application retrieves the same chunks frequently, say a FAQ bot that keeps pulling the same product descriptions, caching those chunks cuts both retrieval costs and token costs. This requires some analysis of your retrieval patterns to identify which chunks appear often enough to be worth caching. ## Limitations worth knowing before you build Prompt caching works well in the right conditions. A few things can limit the savings if you don't account for them upfront. ### Cache TTL Cached prompts don't last forever. On Bedrock, the default time to live (TTL) is around five minutes. For high-frequency applications, the cache stays warm and this isn't an issue. For low-frequency workloads, cache misses happen more often than expected and the savings don't materialize as projected. Factor this into your estimates before assuming a specific hit rate. ### Model support Not every model on Bedrock or Azure OpenAI supports prompt caching. Claude models on Bedrock support it. On Azure OpenAI, caching is available for GPT-4o and a growing list of models. It's worth checking before building your architecture around it. ### Cache miss rates A cache miss means the cached section didn't match and the full prompt got processed at the standard rate. If your prompts vary between calls, through different versions, A/B tests, or personalized instructions, the hit rate drops and the savings shrink. Monitor hit rates once caching is live rather than assuming they'll be high. ### Dynamic prompts Caching only helps with the static parts of a prompt. If most of your prompt is dynamic, with real-time data, per-user context, or live system state, there's less to cache and the benefit is smaller. It's worth auditing your prompts before building caching into the architecture. ## Measuring the impact of prompt caching with CloudSpend Knowing whether caching is working requires visibility into token-level costs that most teams don't have. CloudSpend tracks cost per million tokens per model, so the impact of caching shows up directly in the numbers. If you enable caching on the support bot and the cost per million input tokens drops from $3 to $0.40, that's visible in the dashboard. If the numbers don't move, the cache hit rate is lower than expected and worth investigating. Beyond that, CloudSpend gives you: - **Spend breakdown by team, application, and model**: You can see exactly which workloads are driving input token costs and where caching is likely to have the biggest impact. - **Anomaly detection**: If caching breaks or stops working as expected, input token costs go up. CloudSpend flags the spike automatically across email, Slack, and other channels before it compounds. - **Budget alerts by model**: Set a spend limit on a specific model so if input token costs creep back up after implementing caching, you know about it before the bill lands. ## Author Spotlight ![Elizebeth Joseph](https://cdn.manageengine.com/sites/meweb/images/cloudspend/elizebeth-joseph.png) **Elizebeth Joseph** Product Specialist at CloudSpend Elizebeth is a product specialist at ManageEngine with expertise in FinOps and cloud cost management. She writes about how to solve real spending challenges, from navigating product features to broader strategies for optimizing spend and building cost accountability across teams.