How to reduce AI inference costs on AWS Bedrock and SageMaker
AI is generating real value for engineering teams. But it's also generating real bills, and for most companies, those bills are growing faster than expected.
Inference costs are easy to ignore early on. A few API calls in development, a small spike in testing, it's nothing too alarming. Then you ship to production, usage scales, and you're looking at a five-figure monthly Bedrock bill with no clear idea of where it's coming from.
The instinct is usually to look at the biggest model you're using and swap it out for something cheaper. Sometimes that helps, but it's rarely the whole story. In most cases, overspending on inference comes down to a handful of fixable patterns: Using the wrong model for the task, paying for context you could cache, running real-time calls for workloads that don't need them, and running compute that nobody is actually using.
Fixing these doesn't mean compromising on output quality. It means being more deliberate about how and when you're calling models. Here's where to start.
First, understand what you're actually paying for
Inference is what happens every time your application calls an AI model. iI could be to summarize a document, answer a question, or generate code. Each call costs money; at scale, that adds up fast.
On AWS Bedrock, you pay per token, both input and output. On SageMaker, you pay for the compute running your model, whether or not it's serving traffic.
Both of these options add up quickly. A product making a million Bedrock calls a day at $0.01 per call is spending $300,000 a month. That's before you account for multiple teams, multiple models, and multiple applications all running in the same AWS account.
The other issue is visibility. Most teams don't have a clear breakdown of which application, team, or model is responsible for what. They just see one large AWS bill.
Six ways to cut your AI inference bill on AWS
1. Use the right model for the task
This is the most impactful change most teams can make. Frontier models like Claude Sonnet are genuinely capable, but they're overkill for most production tasks.
Classifying support tickets or extracting structured data from a form? A smaller model can handle that. On Bedrock, switching from Claude Sonnet to Claude Haiku for appropriate tasks can cut costs by 80% on those calls, with no meaningful quality difference for simple workloads. The key is being deliberate about which model each task actually needs.
2. Cache repeated context
Many applications send the same system prompt or context with every request. If your prompt includes a 2,000-token system instruction, you're paying for those tokens on every single call.
Bedrock supports prompt caching. For applications with consistent, repeated context like customer service bots, internal tools, and document processors, this alone can reduce token costs by 30-50%.
3. Use batch inference for non-urgent workloads
Not everything needs a real-time response. Nightly report generation, bulk document processing, data enrichment pipelines, all of these can run asynchronously.
Bedrock Batch is significantly cheaper than on-demand inference. If you're running real-time calls for workloads that could tolerate a few hours of latency, you're leaving money on the table.
4. Review your provisioned throughput
Bedrock lets you reserve throughput in advance at a lower per-token cost, but only if your volume justifies it. Teams that set up provisioned throughput and then reduce usage end up paying for capacity they're not using.
Check whether your actual usage matches what you've provisioned. Often on-demand is cheaper once you factor in idle capacity.
5. Right-size SageMaker instances
If you're hosting custom or fine-tuned models on SageMaker, the instance type matters. Many teams provision large GPU instances during initial setup and never revisit the decision.
Run your actual traffic numbers against smaller instance types. An ml.g5.xlarge is significantly cheaper than an ml.g5.12xlarge, and for many inference workloads, it's sufficient.
6. Find and shut down idle endpoints
This is one of the most common sources of SageMaker waste. An endpoint gets spun up for testing or a proof of concept, traffic drops off, but the instance keeps running.
At $2-5 per hour for a GPU instance, an idle endpoint costs $1,500-3,600 per month. Multiply that across a few forgotten experiments and it adds up fast.
How CloudSpend helps you find these issues
The levers above are straightforward. The hard part is knowing where to look.
Break down your Bedrock spend by model, team, and application
Most teams are working off a single AWS cost line item for Bedrock. It tells you what you spent, but not why. CloudSpend connects to your AWS tags so you can filter spend by model, team, and application from a single dashboard.
That means instead of seeing a $23,000 Bedrock bill, you can see:
- The data enrichment pipeline spent $8,400 on Claude Sonnet last month
- The customer support bot accounts for 55% of total Bedrock spend
- The payments team's usage jumped 80% in the last two weeks
Token-level cost tracking
CloudSpend tracks cost per million tokens for each model, so you can directly compare what different models are actually costing you across workloads. If you move a workload from Claude Sonnet to Haiku, you'll see the cost per million tokens drop in the dashboard, not just a vague reduction in the overall bill.
Alerts before the bill arrives
When spend moves in the wrong direction, CloudSpend can notify you via email, Slack, or several other channels. There are two types of alerts:
- Threshold alerts fire when a team, application, or model crosses a spend limit you define
- Anomaly detection flags unusual patterns automatically, so if a model's usage spikes outside its normal range, you'll know about it without having to set a specific threshold for every scenario
SageMaker visibility
For SageMaker, CloudSpend breaks down spend by endpoint and instance type. Identifying idle endpoints still requires your own monitoring, but knowing exactly what each endpoint costs makes it easier to prioritize where to look first.
CloudSpend gives you the visibility to start
You don't need to implement all six levers at once. Start with visibility. Get a clear picture of what you're spending, on which models, across which teams.
From there, model selection and prompt caching are usually the fastest wins. Right-sizing and idle endpoint cleanup follow once you can see the data.
Most of these changes take days to implement, not months. The harder part is knowing where to look first.
CloudSpend gives you that starting point. Connect your AWS account, and you'll have your Bedrock and SageMaker spend broken down by model, team, and application.
Sinjan covers cloud cost management and FinOps at ManageEngine. He writes about the financial realities of running cloud infrastructure, from optimizing cloud spend to fostering cost accountability across teams, helping organizations move from cloud cost chaos to confident, data-driven decisions.