But cloud waste isn't going down because of it. If anything, it's climbing, one estimate puts it at 29% in 2026, almost entirely because of AI workloads that don't behave the way older infrastructure monitoring expects them to. So teams are spending more than ever on observability and still getting caught off guard. In this article we will discuss what AI observability needs to cover, why most teams are only watching half of it, and what it looks like to close that gap.
What AI observability actually means
At its core, AI observability just means knowing what your AI system is doing, why it's doing that, and what it's costing you, in production, all the time. The concept grew out of regular software observability (logs, metrics, traces), but AI systems break in ways that regular code doesn't.
A normal service either returns the right answer or it doesn't, and when it fails, you get a stack trace. AI doesn't fail that cleanly. A model can hand back a confident, well-written, completely wrong answer. An agent can finish every step of a task correctly and still solve the wrong problem. None of that trips an uptime alert. That's the gap ML observability was originally built to close, watching for data drift, concept drift, accuracy quietly slipping over time.
Then LLMs and agents added a whole new set of failure modes on top: hallucinations, prompts behaving unpredictably, drift in embeddings that doesn't show up as a normal statistical shift, multi-step reasoning that goes sideways in ways no confusion matrix was built to catch. This is what most people picture when they hear "AI observability" today. It's a real and necessary piece of the puzzle. It's just not the whole puzzle.
Why so much observability spend still leaves teams blindsided
Part of the problem is that ML observability tools were simply never built to look at infrastructure. A tracing platform can tell you a model's outputs are accurate and safe. It has no idea whether the GPU running that model is sitting at 15% utilization or 95%, because that was never what it was designed to check. If your team treats model-quality tooling as the entirety of "AI observability," there's a whole category of failure nobody's watching.
Kubernetes doesn't help here, either. It hands out an entire GPU to a pod the moment that pod asks for one, whether the workload actually needs 90% of that device or 10%. Studies of production clusters running lightweight inference jobs have found utilization sitting in the single digits. That's not a rare misconfiguration; it's closer to the default. And no model-quality dashboard will ever flag it, because it isn't a model problem.
The last piece is that cost spikes rarely look like anything is wrong. A retry loop, an agent that starts calling itself in circles, a multi-GPU instance nobody remembered to shut down over the weekend, all of these can rack up real money without a single output ever being flagged as incorrect. The system is doing exactly what it was told to do. It's just an expensive way to do it, and most observability stacks aren't set up to notice.
The three layers a real practice actually needs
If you break observability ai down into what it's actually made of, there are three separate jobs, and none of them substitutes for the other two:
The first is watching model behavior
Tracing what an agent or pipeline does step by step, evaluating outputs against a golden dataset, tracking drift, catching hallucinations before a user sees them. This tells you whether the system is doing the right thing, and it usually sits with ML and product teams.
The second is infrastructure
GPU utilization, what's actually happening inside your Kubernetes clusters, how autoscaling is behaving, whether the serving layer underneath the model is healthy. This tells you whether the system is running efficiently on the hardware you're paying for, and it's usually owned by platform or cloud engineering, the team that doesn't get paged when something overspends quietly instead of breaking loudly.
The third is cost
GPU hours, tokens, provider bills, all tied back to a specific team, feature, or customer. This tells you what the system actually costs and whether that's reasonable given what it's producing. This is the layer most teams don't have anyone clearly owning, because it sits in the awkward space between engineering and finance.
A team with great tracing and no utilization visibility will build a system that works fine and quietly drains the budget. A team with great cost dashboards and no model-quality tracking will keep spend under control on a system they can't actually vouch for. You need all three, or you're just choosing which blind spot you're comfortable with.
How to tell you've got a blind spot
A few things tend to show up before anyone officially notices there's a problem. Some of them are:
- A GPU cluster that's been reserved for months, and nobody can say, without going and checking, what its utilization actually looked like this week.
- A monthly bill that swings for no reason tied to actual traffic.
- An eval dashboard showing green across the board while nobody on the team can tell you what a single customer or feature actually costs to serve.
- GPU instances that keep running after a job finishes, caught days later instead of minutes later.
- An agent that's hitting every one of its success metrics while quietly making far more tool calls, and burning far more tokens, per task than anyone budgeted for.
None of that is a model problem. It's an infrastructure and cost problem wearing a model-shaped disguise, and by the time it surfaces, it's usually already expensive.
What to actually measure
Most teams end up tracking whatever their eval tool shows by default: accuracy, latency, some drift score, mostly because that's what came pre-configured. That's fine as far as it goes, but it's a small slice of what a real ML observability setup should be watching.
On the infrastructure side, you want GPU utilization broken out by memory and by compute (a workload can be starved on one and fine on the other), idle time per instance, and how much of a cluster's total capacity is actually doing work versus just sitting reserved. On the cost side, you want spends broken down by model, team, feature, and customer, anomalies flagged against a rolling baseline instead of a flat threshold, and unit economics: cost per inference, cost per active workload — instead of one number at the bottom of an invoice.
That last one tends to matter more than people expect, because it's the difference between "our AI spend went up" and "our AI spend went up faster than the usage that's supposed to be driving it." The second version is the one that actually holds up in a budget meeting.
Questions Every Team Should Be Asking About Their AI Infrastructure
As AI projects rise, it’s not enough to know if a model is generating accurate results. Teams must now also consider whether the infrastructure supporting these models is performing efficiently. Asking yourself the following questions will help find any blind spots before they become costly.
1. Are your GPUs efficiently allocated?
Just because your model has a GPU doesn’t mean that it is using it efficiently. Watch the GPU compute and memory utilization to spot any over-provisioned or idle workloads. A small efficiency boost could save you huge sums in your cloud budget.
2. Is it possible for you to track the cost of AI per team, per feature or per customer?
Your cloud bill doesn’t give you insights into where costs come from. Breaking down your spend per workload, per application, per customer or per business unit will help you understand which initiatives generate value and which are worth optimizing.
3. Are your AI agents becoming more expensive overtime?
AI agents tend to be more efficient in calling tools, consuming tokens and executing more sophisticated workflows over time. Watching cost per task or cost per inference will help you catch inefficiencies early on.
4. Is infrastructure scaling in proportion to demand, or merely remaining provisioned?
Infrastructure should be autoscaled in accordance with the real load, and not in accordance with the peak load that was experienced some days or weeks ago. Ensure that the unused GPU instances, underutilized clusters, and idle infrastructure are periodically checked to make sure you are not paying for unused infrastructure.
5. Do engineering and finance teams have a common understanding?
While the engineering team is concerned about the performance of the AI operations, the finance team is interested in spending. Aligning the infrastructure metrics and cost metrics will provide you with an insight into the operation of your AI processes.
6. Are you tracking efficiency as well as performance?
Accuracy, latency, and response are certainly important, but they tell only part of the story. An advanced AI operations system will also have such efficiency metrics as cost per inference, GPU utilization rate, token usage, idleness, and efficiency of the workloads.
Putting together a stack that actually covers all of it
In practice, a workable setup usually combines a tracing and evaluation tool for model behavior, drift detection that's actually built for LLMs (not repurposed from classic ML, since the failure mode is semantic rather than statistical), Kubernetes-level visibility into GPU and workload usage that goes deeper than whatever your cloud provider's billing console shows you, cost data that's mapped to a team or feature rather than just a line item, and anomaly detection that fires on spend and utilization, not just on broken outputs.
None of these pieces do much on their own. A drift alert next to a cost spike tells you a lot more than either one alone and teams running both usually get to the root cause a lot faster than teams running just one.
The two kinds of tools, and why they're not the same thing
The AI observability market has really split into two camps, and it's worth knowing which one you're actually buying. LLM tracing and eval platforms — most of the newer, LLM-native vendors — handle correctness: tracing, evaluation, hallucination detection, drift in prompts and embeddings. If you're running agents in production, you need this.
Cost and infrastructure platforms are a different animal entirely. They handle GPU utilization, Kubernetes cost allocation, and spend across providers. The buyer here is usually cloud engineering or FinOps, not the ML team, and this is the layer most companies haven't gotten around to buying yet — not because it matters less, but because it wasn't the fire that was burning first.
Most mature teams end up with one of each, the same way you'd want both a debugger and a profiler. One tells you the code is right. The other tells you what it costs to keep it that way.
Where teams usually go wrong
A few patterns show up again and again. Teams assume their tracing and eval tool is observability ai in full, so cost and utilization problems drift for months before anyone notices. They rely on their cloud provider's default billing dashboard, which almost never breaks costs down by namespace or workload in a way that's actually useful. They let agents call themselves or revisit context with no ceiling, so one misconfigured workflow quietly eats a chunk of the monthly budget. They alert on errors but not on utilization, so an idle GPU cluster can burn cash for weeks without tripping anything. And nobody owns the infrastructure and cost layer specifically, because it falls in the gap between engineering and finance, which is exactly why it's the one that gets missed.
How OneLens by Astuto fits in
OneLens focuses specifically on the layer most ML observability stacks skip: infrastructure and cost. It pulls together visibility across your cloud infrastructure, Kubernetes clusters, GPU usage, and spend across AI providers — OpenAI, Anthropic, AWS Bedrock, Azure AI Foundry, Google Vertex AI — so you're not stitching that picture together across five different billing dashboards.
It tracks GPU and cluster utilization down to the workload level, so idle capacity gets flagged automatically instead of sitting there until the bill arrives. Spend gets tied back to the team, feature, or customer generating it, so a lump-sum invoice actually turns into something you can act on. And anomaly alerts go to the people who own that spend, so a runaway agent or a forgotten instance gets caught in hours instead of showing up as a surprise next month. If you've already got model-quality observability AI in place, this is usually the piece that's missing — the one that tells you not just whether the model is behaving, but whether the infrastructure and cost behind it are too.
Conclusion
All the money going into AI observability right now is solving a real problem, model behavior genuinely does need to be traced and evaluated in production. But it's only solving part of it. Cloud waste climbing in 2026, in the middle of record observability spending, isn't proof that the tooling doesn't work. It's proof that most of it is watching one layer while an equally expensive one runs unmonitored right underneath it.
A complete practice treats model behavior, infrastructure, and cost as three separate things that each need an owner, not one thing that happens to cover the other two. The teams that are ahead on this right now aren't the ones who've bought the most tools. They're the ones whose tooling actually covers all three layers, so nothing about how their AI runs, or what it costs, catches them off guard.
.jpeg)
