Defining Reliability for AI Systems
Why uptime is not enough when outputs can be available, fluent and wrong.
By Mark Bourne
Introduction
Ask an engineering team whether their AI system is reliable, and most will point at a dashboard.
Green status. Low latency. No alerts. Uptime at 99.9%.
By every measure that used to matter, the system is healthy.
And it can still be quietly wrong, hour after hour, without a single alert firing.
That gap is the subject of this article. Reliability engineering has a century of practice behind it for systems that fail in binary, observable ways—a server is up or down, a packet arrives or it doesn't. AI systems fail differently. They stay up. They respond promptly. They sound confident. And the answer can still be false.
Available, fluent and wrong is not a degraded state. It is a failure mode that looks exactly like success on every chart that currently exists.
Internet Architecture Lessons for AI Systems argued that reliability matters more than occasional brilliance. This article takes the next step: turning that instinct into something measurable.
The Internet Lesson: Uptime Was Never the Whole Story
Even in networking, uptime alone was always an incomplete metric.
A link can be up while dropping packets. A server can respond while serving stale data. A DNS resolver can answer instantly with the wrong address.
That is why mature Internet operations never relied on availability by itself. They measured packet loss, error rates, resolution correctness, and latency distributions alongside it.
AI systems need the same discipline—but the failure surface is larger, because the “wrong answer” problem is not an edge case. It is the default behaviour of a model that has no concept of being wrong.
A Realistic Failure Scenario
Picture a support assistant answering billing questions from a knowledge base.
A pricing document is updated. The retrieval index isn't refreshed for six days.
For those six days, the assistant answers instantly, in a friendly and confident tone, citing a document that no longer reflects reality.
Every operational signal stays green:
- Uptime: 100%
- Latency: well within budget
- Error rate: zero
- Support tickets: nothing unusual, yet
Nothing crashed. Nothing timed out. Nothing looked like an incident—until the refund requests started arriving, days later, for a problem no dashboard had shown.
That is the scenario this article is trying to make impossible to miss.
Two Kinds of Failure
Classic systems mostly fail deterministically: the same bad input produces the same crash, every time. Fix the bug, the failure disappears.
AI systems introduce a second category: non-deterministic failure. The same prompt, sent twice, can succeed once and fail once. Nothing about the code changed.
- Deterministic: timeouts, malformed requests, API outages, invalid tool calls
- Non-deterministic: hallucination, inconsistent reasoning, drifting tone, silent omission
Deterministic failures are debugged. Non-deterministic failures are managed statistically—which means reliability targets for AI have to be expressed as rates and distributions, not as pass/fail conditions.
Accuracy, Consistency and Completion Are Not the Same Thing
Teams often collapse quality into a single number. That single number hides which problem is actually occurring.
- Accuracy: is this specific answer correct?
- Consistency: does quality hold steady across repeated, similar requests?
- Completion: does the workflow finish end to end, or abandon partway through?
A system can be highly accurate but wildly inconsistent—brilliant on Monday, unreliable on Tuesday. Another can be consistent but consistently mediocre. A third can be accurate and consistent yet fail to complete a third of workflows because a tool call silently stalls. Each of these needs a different fix, so each needs its own number.
Latency Means Something Different Now
Web latency budgets were built around a single round trip. AI workflows often chain several: retrieval, one or more model calls, tool execution, validation.
Users tolerate this differently depending on what they see. A streaming first token in under a second feels fast, even if the full response takes twenty. A silent ten-second wait for a single blob of text feels broken, even if it was actually faster overall.
Reliability targets should therefore separate:
- Time to first token
- Time to full completion
- Tail latency (p95/p99), not just the average
- Timeout behaviour when a chained step stalls
AI Service-Level Indicators
Traditional SRE practice defines Service-Level Indicators before setting targets. AI systems need an expanded set, because “the service responded” and “the service was useful” are entirely different claims.
| Indicator | What it actually measures |
|---|---|
| Availability | Is the service reachable and responding at all? |
| Latency | How long until the first token, and until completion? |
| Accuracy | Is the output factually and logically correct? |
| Consistency | Does the same input produce comparable quality every time? |
| Completion rate | Does the workflow finish, or stall partway through? |
| Groundedness | Is the answer supported by retrieved or cited evidence? |
| Escalation rate | How often does a human need to step in? |
| Cost per success | What does one completed, correct outcome actually cost? |
Not every indicator matters equally for every workflow. A drafting assistant can tolerate lower groundedness than a system quoting refund policy. Choosing which indicators matter is itself an architectural decision.
Defining Acceptable Failure
“Reliable” does not mean “never wrong.” No AI system clears that bar, and chasing it stalls shipping indefinitely.
The useful question is narrower:
For this specific workflow, what failure rate is acceptable, and what happens when a failure occurs?
A summarisation tool that occasionally drops a minor detail is a different risk profile than a system approving financial transactions. The acceptable failure rate, and the required blast radius of each failure, should be written down before launch—not discovered from an incident report.
Error Budgets for AI Workflows
Site reliability engineering popularised the error budget: decide how much unreliability is tolerable, then spend it deliberately on velocity, rather than treating every failure as a crisis.
The same idea applies to AI workflows:
- Define a target success rate for the workflow, not the model
- Track budget burn the same way for a wrong answer as for an outage
- Slow down releases when the budget is exhausted early
- Treat a fast burn as a signal to investigate, not a reason to panic silently
An error budget turns 'is it good enough?' into a number.
Numbers can be tracked. Feelings can't.
Measure Usefulness, Not Intelligence
Benchmark leaderboards measure how a model performs on curated tasks. They rarely predict how a workflow performs on your data, your users, and your failure tolerances.
A smaller, cheaper model wired into a well-instrumented workflow will often outperform a frontier model bolted onto an unmeasured one.
Reliability is a property of the system, not a property of the model.
Trade-offs and Anti-Patterns
A few patterns show up repeatedly once teams start measuring this seriously:
- Chasing 100% accuracy instead of shipping at an acceptable, monitored rate
- Treating every wrong answer as a page-worthy incident, which trains teams to ignore alerts
- Equating 'the API returned 200' with 'the answer was correct'
- Averaging latency without ever looking at the tail
- Optimising the model in isolation while the surrounding workflow stays unmeasured
Each of these substitutes an easy number for the hard one. The easy number always looks better on a dashboard, right up until it doesn't.
Final Thoughts
Networks earned trust by making failure visible, measurable, and survivable. AI systems have to earn the same trust, against a harder problem: failures that look identical to success unless someone defined, in advance, what success actually means.
Uptime tells you the system responded.
Only a defined standard of reliability tells you whether it should have.
That standard has to be written before the incident, not reconstructed after it.
A Practical Checklist
Before calling a workflow production-ready, work through these questions:
- 1Have you written down, in one sentence, what a 'successful' output looks like for this workflow?
- 2Do you track accuracy and consistency separately, rather than one blended 'quality' score?
- 3Is there a defined latency budget, including tail latency, not just an average?
- 4Do dashboards distinguish 'the model responded' from 'the model was right'?
- 5Is there an error budget, with an agreed response when it's exhausted?
- 6Do you know your current escalation rate, and whether it's trending up or down?
- 7Is cost measured per completed, correct outcome rather than per API call?
- 8Would a silent week of confidently wrong answers actually get noticed?
About This Series
This article is part of the AI Infrastructure & Architecture series on Bourne Forge AI, where decades of Internet architecture experience are applied to modern AI systems. This piece turns the reliability principles introduced in Internet Architecture Lessons for AI Systems into a measurable engineering discipline, as groundwork for the articles that follow on failure handling, observability, and context architecture.
More from the Notes
Short technical notes and observations, written up as experiments produce something worth documenting.
Was this useful?
Published