Insights
What we have learned keeping AI running.
Not commentary on the industry. The specific things that break, what they cost, and what we would tell you not to build — written from operating our own AI platform in production and from the engagements that came out of it.
What actually breaks an AI agent in production
The failure modes, in the order we have met them — and almost none of them are the model being wrong.
Read itThe short answer
AI agents rarely fail because the model gives a wrong answer. They fail because the retrieval layer goes stale, because a provider deprecates the model underneath them, because nobody defined what the agent should do when it does not know, because costs scale in a way nobody modelled, and because no one is watching any of it after launch. In production the model is the most reliable component in the system. Everything around it is what breaks.
How to cost an AI automation before you build it
Four lines of arithmetic that decide whether a project is worth funding — and that most proposals skip.
Read itThe short answer
To cost an AI automation, work out what the current process costs before you look at what the system costs. Multiply volume by minutes per unit by the fully loaded hourly cost, then add rework: error rate times the time to fix. That is your annual baseline. Against it, put the build cost plus twelve months of running cost, where running cost includes model and infrastructure spend, monitoring, and the engineering time to keep it accurate. If the baseline does not comfortably exceed the total, the honest answer is not to build it.
Why your RAG system should refuse more often
A retrieval system that never says “I don’t know” is not accurate. It is unmeasured.
Read itThe short answer
A retrieval-augmented system should refuse whenever it cannot ground an answer in a retrieved source, and a healthy one refuses more often than teams expect. Refusal is not a failure state — it is the behaviour that makes every other answer trustworthy. If a system never declines, it is not that the corpus covers everything; it is that the system will always produce something, and nobody has measured how often that something is unsupported.
What changes when buyers ask ChatGPT instead of Google
AI answers name one or two companies and send no click. Your analytics cannot see any of it.
Read itThe short answer
When a buyer researches inside an AI engine rather than a search engine, they get a synthesised answer naming one or two companies rather than a page of links — and no click is sent, so the interaction never appears in your analytics. Being cited depends on whether an engine can parse your content, verify your claims and identify you as an entity, which is a structural property of your site rather than a ranking you can bid for. The practical consequence is that a growing share of consideration now happens somewhere you cannot currently measure.
Why we publish this
It is the only proof we can offer before you hire us.
We do not publish client logos we cannot evidence, and a credentials deck proves nothing. What we can do is show our working — how we cost a build, what we monitor, where we would tell you not to spend.
Written from operating, not observing
Every failure mode described here is one we have had to fix, most of them on our own platform before any client saw them.
The arithmetic is yours to run
Where a number appears it is a formula you can put your own figures into, not an industry average from a slide.
Including the parts that lose us work
A third of the automations we assess do not pay for themselves. Saying so publicly costs us projects and is the reason the rest are worth having.
Rather see it than read about it?
Tell us the problem and we build a working demo on your own material within 48 hours. Free, yours to keep, and considerably more informative than any article.