What actually breaks an AI agent in production
The failure modes, in the order we have met them — and almost none of them are the model being wrong.
The short answer
AI agents rarely fail because the model gives a wrong answer. They fail because the retrieval layer goes stale, because a provider deprecates the model underneath them, because nobody defined what the agent should do when it does not know, because costs scale in a way nobody modelled, and because no one is watching any of it after launch. In production the model is the most reliable component in the system. Everything around it is what breaks.
The retrieval layer goes stale before anything else
The most common failure we see is not a hallucination. It is an agent answering correctly from a document that stopped being true three weeks ago.
A retrieval system is only as current as its last index. Teams build the pipeline, load the corpus, test it against that corpus, and ship. Then a price changes, a policy is superseded, a product is discontinued — and the agent keeps citing the old source with complete confidence. It is not hallucinating. It is being accurate about something that no longer holds.
This is a plumbing problem rather than an AI problem, which is exactly why it gets skipped. Re-indexing on a schedule is unglamorous work that nobody demos.
Ask any vendor how their retrieval stays current, and what happens on the day the source changes. The answer tells you whether they have run one of these for a year.
The model underneath you is deprecated on somebody else’s schedule
Providers retire model versions. That is normal, it is announced, and it is still the single most disruptive event in the life of a production AI system — because behaviour changes in ways no release note can fully describe.
Without an evaluation suite, a migration is a gamble: you swap the model, the obvious cases still work, and you discover the regression from a customer weeks later. With one, you re-run it, see exactly which categories moved and by how much, and decide deliberately.
The suite is not there to prove the system is good. It is there to tell you what changed.
Nobody designed what happens when the agent does not know
Most agent builds specify the happy path in detail and leave the unhappy path to the model’s discretion. That is backwards. The happy path is the part that works by default.
An agent with no designed refusal behaviour will produce something plausible rather than nothing, because that is what the objective rewards. In a retail context that becomes an invented delivery date. In a regulated one it becomes advice you are not licensed to give.
Refusal has to be specified, tested and measured like any other behaviour: what triggers it, what it says, where it escalates, and how often it fires. An agent that never refuses is not confident. It is unmonitored.
Cost scales in a shape nobody modelled
Per-call cost is easy to reason about and almost never the problem. The problem is that context grows: conversation history accumulates, retrieval returns more chunks, someone adds a longer system prompt, and the same conversation costs several times what it did at launch.
Unit economics have to be modelled before code, then instrumented after. Caching, routing cheaper models at the right steps, right-sizing context, and a hard ceiling that alerts rather than silently bills.
This is also why margin on token usage is a conflict of interest. A vendor who profits from your consumption has no reason to make any of those decisions in your favour.
Integration surfaces move quietly
The agent depends on your CRM, your commerce platform, your ERP, your telephony. Those systems get upgraded, fields get renamed, an API version is retired, someone changes a permission.
None of this is exotic. It is ordinary software entropy, and it is why an AI system needs the same monitoring discipline as any other integration — not the special treatment AI projects usually get, where they are either a science experiment or an untouchable black box.
And then nobody is watching
Every failure above is detectable. Accuracy against a held-out set, refusal rate, escalation reasons, cost per conversation, retrieval freshness, latency. All of it can be measured continuously and almost none of it usually is.
The pattern is consistent: an AI project is resourced heavily through launch and not at all afterwards. But the value of the system is decided in the eighteen months after launch, not the eight weeks before it.
What to ask before you sign anything
- How does the retrieval layer stay current, and what happens the day a source document changes?
- Show me the eval suite. What does it cover, and what happens when a provider deprecates the model?
- What does the system do when it does not know? Show me it refusing.
- What is the cost per conversation, and what happens when context grows?
- What is monitored after launch, who sees it, and what triggers someone to act?
A vendor who can answer all five has run one of these before. A vendor who answers the first and deflects the rest has built a demo.
Questions this raises
Is hallucination not the main risk with AI agents?
It is the most discussed risk and rarely the one that causes production incidents. Grounded retrieval with citation enforcement and designed refusal handles most of it. The failures that actually reach customers tend to be stale data, an undesigned unhappy path, or a model migration nobody tested — all of which are engineering problems rather than model problems.
How often do model deprecations actually happen?
Often enough to plan for rather than react to. Providers retire versions on their own schedule with notice, so the question is not whether it happens but whether you find out from a release note or from a customer. An eval suite is what makes the difference between a planned migration and an incident.
What does monitoring an AI system actually involve?
Accuracy against a held-out set, refusal and escalation rates with reasons, cost per transaction against a ceiling, retrieval freshness, and latency. Reported on a cadence with a named owner. If nobody is accountable for reading it, it is not monitoring.
This is what we do about it
More like this
How to cost an AI automation before you build it
Four lines of arithmetic that decide whether a project is worth funding — and that most proposals skip.
Why your RAG system should refuse more often
A retrieval system that never says “I don’t know” is not accurate. It is unmeasured.
What changes when buyers ask ChatGPT instead of Google
AI answers name one or two companies and send no click. Your analytics cannot see any of it.