By most measures of computing history, that should have made AI cheaper to run. Instead, the opposite happened at company after company. Uber's engineering organization burned through its entire annual AI coding budget in just four months in 2026. Microsoft pulled back internal developer access to a popular coding assistant over the same problem. Neither team was being careless, they were running into the same math now showing up on finance dashboards everywhere: per-token prices keep falling, and the bill keeps climbing anyway.
That contradiction is the whole story of AI inference cost right now, and it's why it's become one of the fastest-growing, least-understood line items to defend. This piece breaks down what's actually driving that gap, what inference cost is made of underneath the token price, and the concrete moves that bring it back under control without gutting model quality.
What AI Inference Cost Actually Is
AI inference cost is the money spent running a trained model to generate a response, every prompt processed, every token produced, every API call or GPU-second consumed once a model is live and serving real traffic. It's distinct from training cost, which is a one-time or occasional expense to build the model in the first place. Inference doesn't have a natural stopping point. It runs for as long as a feature is live, scaling with every user, every request, every retry.
That distinction matters more than it used to. Training used to be where most of an AI budget went. Not anymore, inference now accounts for somewhere between 55% and 90% of enterprise AI GPU spend, depending on whose benchmark you use, and the FinOps Foundation's 2026 industry survey puts the majority of that range on the higher end. A cost model still built around the training run is missing most of where the money actually goes.
Why Per-Token Prices Keep Falling
Competition between OpenAI, Anthropic, Google, and a growing list of specialized inference providers has pushed prices down consistently. Hardware keeps improving, model architectures have gotten more efficient at converting compute into capability. Stanford's AI Index tracked roughly a 280-fold drop in the price of GPT-3.5-equivalent capability between late 2022 and late 2024, and Gartner's 2026 forecast expects frontier-tier inference to keep getting cheaper, potentially another 90% reduction by 2030.
None of that is in dispute. It's also, on its own, not translating into lower bills for the companies actually running these models in production.
Why the Bill Keeps Rising Anyway
Three things explain the gap, and they compound:
Agentic workflows multiply requests, not just tokens
A single agentic customer-service interaction that cost roughly four cents in 2023 costs around $1.20 today, according to EY's analysis , a thirty-fold increase driven by agents that reason, call tools, and revisit context multiple times per task instead of generating one response and stopping. Every one of those intermediate steps is a billable inference call. A workload that looks the same from a user's perspective can now generate ten or twenty times the token volume behind the scenes.
Reasoning models think longer before they answer:
Newer models that work through a problem step by step before producing an output consume substantially more tokens per response than earlier chat-style models, even when the visible answer looks identical in length. That extra reasoning happens inside the token count, which means it shows up on the bill whether anyone budgeted for it or not.
Volume grows faster than price falls:
Once AI moves from pilot to core operation, which, per recent industry reporting, describes the large majority of enterprises by mid-2026, usage stops being occasional and starts being constant. A 90% drop in per-token price doesn't help much if request volume grows 10x in the same period. The math nets out to a bigger AI Inference Cost, not a smaller one, and it's why average enterprise AI budgets have grown from roughly $1.2 million a year in 2024 to something closer to $7 million in 2026 at the organizations tracking this closely.
What Actually Makes Up an Inference Bill
Beneath the sticker price of "$X per million tokens," a few specific factors determine what any given workload actually costs to run:
- Input and output token volume: Most providers have a higherAI Inference Cost for output tokens than input tokens, since generating text is more compute-intensive than reading it. A verbose model, or a prompt that invites a long answer, costs more per exchange even at an identical per-token rate.
- Context window size: Every token in the context, conversation history, retrieved documents, system instructions, gets reprocessed on each call unless something is caching it. Long context is convenient and expensive at the same time.
- Model size and choice: Larger, more capable models cost more per token to run than smaller ones, sometimes by an order of magnitude, which is exactly why routing decisions matter as much as raw pricing.
- Batching and GPU utilization: For self-hosted inference, how efficiently requests get grouped onto a GPU has a direct effect on cost per request, a poorly batched deployment can cost several times more than a well-tuned one running identical hardware.
- Provider markup and infrastructure overhead: Cloud-hosted API access typically carries a margin over raw compute cost, in exchange for reliability, compliance, and not having to run the serving infrastructure yourself. That markup is usually worth paying for unpredictable or low-volume workloads, and worth questioning once a workload's traffic becomes steady enough to justify a different arrangement.
- Retry and failure rates: A request that times out, gets a malformed response, or needs a follow-up call to clarify still consumed tokens the first time. High retry rates on a workload are an AI Inference Cost problem hiding behind what looks like a reliability problem.
Signs Inference Cost Has Gotten Away From You
A few patterns tend to show up before anyone officially raises the alarm. A feature that shipped with a modest usage estimate is now generating requests nobody explicitly greenlit at that volume, an agent quietly calling itself in a loop, or a retry policy firing more often than intended.
The monthly bill has a wide, unexplained swing between weeks with no corresponding change in user traffic. Every request, simple or complex, is still hitting the same expensive frontier model because nobody's built a cheaper path for the easy ones. Or the team can quote a per-token rate confidently but can't answer what a single customer interaction, end to end, actually costs. Any of these means the AI Inference Cost is currently being discovered rather than managed.
Measuring the Right Number
Most teams default to tracking cost per token, mostly because it's the number the provider hands them. It's also not that useful on its own. Two workloads can have an identical cost-per-token rate and completely different economics if one needs three retries to get a usable answer and the other gets it right the first time.
The more useful measure is cost per successful outcome, cost per resolved support ticket, per completed task, per accepted code suggestion, rather than cost per raw token or per API call. A cheaper model that needs twice as many attempts to produce a usable result isn't actually cheaper. Framing inference spend around outcomes rather than tokens is what turns a cost conversation into a business conversation, and it's usually the difference between a team that can defend its AI budget in a review and one that can only describe it.
The Optimization Stack That Actually Moves the Number
Most advice on this topic pitches one technique as the fix. In practice, the teams that meaningfully cut inference spend stack several of these together, since each one attacks a different part of the bill.
Prompt and context caching: Repeated context , a system prompt, a knowledge base excerpt, a long conversation history, doesn't need to be reprocessed from scratch on every call if it's cached. This alone often removes a meaningful chunk of prefill cost on any workload with repeated or shared context.
Batching: Grouping multiple requests into a single GPU pass keeps hardware busier per dollar spent. Continuous batching, now standard in serving frameworks like vLLM, can deliver several times the throughput of naive request-by-request processing, which translates directly into lower cost per request for self-hosted deployments.
Model routing: Not every request needs the most capable model available. Sending simple classification, extraction, or lookup tasks to a smaller, cheaper model and reserving the frontier model for genuinely complex reasoning is consistently one of the highest-leverage levers available, analysis of billions of enterprise API calls found organizations running a tiered model architecture paying a fraction of what organizations routing everything to a frontier model paid, for comparable workloads.
Quantization: Running a model at lower numerical precision shrinks its memory footprint substantially with a small, usually acceptable, quality trade-off. For self-hosted models, this is often the first optimization worth trying, since it costs nothing to train and pays off immediately.
Context and prompt engineering: Trimming unnecessary history, summarizing instead of forwarding full documents, and tightening system prompts reduces token volume at the source, the cheapest optimization of all, because it needs no new infrastructure, just discipline.
Stacked together, these techniques compound rather than simply add up, and production teams applying several at once have reported cutting AI Inference Cost by well over half without a noticeable drop in output quality. One documented case took a 70-billion-parameter model deployment from roughly $39,000 to $16,000 a month using exactly this combination.
Self-Hosting vs. API Access: Where the Line Actually Sits
Whether to call a hosted model API or run inference on owned or reserved infrastructure comes down almost entirely to sustained volume. Below a rough threshold of around 500,000 tokens a day of consistent load, API access to a smaller or mid-tier model tends to beat the cost and operational overhead of self-hosting. Above that threshold, and especially with a workload stable enough to justify dedicated infrastructure, self-hosting starts to close the gap and can undercut API pricing meaningfully, provided utilization stays high, since idle self-hosted capacity is exactly as wasteful as an idle owned GPU cluster anywhere else.
Most organizations end up running both: API access for exploratory or low-volume features, self-hosted or reserved infrastructure for the handful of high-traffic workloads that justify the operational investment.
Common Mistakes That Inflate Inference Cost
Below are some common mistakes companies repeat that pile up the cost on their AI Inference bill. The below factors must be avoided to keep the AI operations lean and cost affective.
Treating the per-token price as the whole story: A falling rate card doesn't mean a falling bill once volume, context size, and agentic multipliers are factored in , and budgeting as if it does is how teams end up surprised.
Sending every request to the same expensive model: Without a routing layer, simple and complex tasks cost the same, which means the bulk of spend goes toward capability that most requests never actually need.
No cap on agentic fan-out: A workflow that can call itself, call tools, or revisit context in a loop needs a hard limit on how far that can go. Without one, a single misconfigured agent can generate a meaningful share of a monthly budget before anyone notices.
Ignoring context window cost: Feeding a model unnecessary history or oversized documents on every single call is one of the most common and most fixable sources of inflated inference spend.
No visibility below the total bill: A monthly invoice that shows one number, with no breakdown by feature, team, or customer, makes it impossible to tell whether an expensive month reflects healthy growth or a problem worth fixing.
Optimizing for cost per token instead of cost per outcome: Switching to a cheaper model that needs more retries, longer prompts, or more follow-up calls to get a usable answer can quietly cost more overall than staying on the pricier one, a trade that only shows up if cost is measured against outcomes rather than raw token rates.
How OneLens by Astuto Helps
OneLens gives engineering and finance teams a clear, per-workload view of AI inference cost, across OpenAI, Anthropic, AWS Bedrock, Azure AI Foundry, Google Vertex AI, and self-hosted models, instead of a single lump-sum invoice at the end of the month.
It tracks token consumption and cost down to the feature, team, or customer generating it, so a spike from a new agentic workflow or a runaway retry loop gets caught in hours, not discovered on next month's bill. Budgets, anomaly alerts, and model-routing recommendations are tied to the people who actually own that spend, turning inference cost from a number that shows up after the fact into one that engineering leadership can see coming and act on before it does.
Conclusion
Falling token prices were never going to be the thing that kept AI bills in check , volume, context, and agentic complexity were always going to grow faster than the price per token could fall. The organizations getting ahead of this aren't waiting for cheaper models. They're building the visibility and routing discipline to make sure every token spent is one that needs to be spent, which is the only version of "cheaper AI" that actually shows up on an invoice.
.jpeg)
