Observability for Non-Deterministic Systems
How to understand an AI application when identical inputs do not guarantee identical outputs.
By Mark Bourne
Introduction
Designing AI Systems for Failure assumed you could detect a failure in the first place. That assumption deserves its own article.
Traditional observability was built on an assumption so basic it was never stated aloud: the same input produces the same output. If it didn't, that was a bug. You found the line of code, fixed it, and moved on.
AI systems break that assumption on purpose.
Send the same prompt twice and you can get two different answers, both defensible, both “correct” in the sense that neither one crashed. Traditional monitoring has nothing to say about that gap, because it was never designed to.
When identical inputs can produce different outputs, “it's working” and “it's working well” stop being the same question.
This article covers what to log, trace, sample and alert on when correctness itself is a statistical property rather than a boolean one.
The Internet Lesson: “What Do the Logs Say?”
The first question in any network incident has always been the same one.
What do the logs say?
That question only works because decades of network engineering invested in making the answer available: structured logs, distributed tracing, metrics with enough context to reconstruct what happened, not just that something happened.
AI teams frequently skip this. A single “model call succeeded” log line tells you the request completed. It tells you nothing about whether the retrieved context was relevant, whether the answer matched the source material, or whether the user immediately corrected it.
An AI system without this depth of logging isn't hard to improve. It's impossible to improve, because nobody can see what's actually happening.
A Realistic Failure Scenario
A support assistant's answer quality quietly drifts downward over three weeks. No single day looks alarming. Latency is flat. Error rate is flat. Uptime is perfect.
The actual cause: a routine prompt edit shipped without versioning, three weeks ago, slightly changed how the model weighed conflicting sources. Nobody logged which prompt version served which response, so there is no way to correlate the edit with the decline.
By the time someone notices—from a spike in support escalations, not a dashboard—three weeks of responses are unrecoverable evidence. Nobody can say exactly when it started or what changed.
The failure wasn't the prompt edit. It was the absence of anything that could have caught it.
Version Everything That Can Change the Answer
In conventional software, the deployed code is the whole story. In an AI workflow, the code is only one of several things that determine the output.
- Prompt and system-instruction text
- Model identifier and provider-side configuration
- Retrieval index version and embedding model
- Tool definitions and their schemas
Any of these can change behaviour without a code deploy. If none of them are versioned and logged alongside the response, a quality regression becomes unattributable—you know something changed, and no way to know what.
Tracing Across the Workflow
An AI workflow is rarely one call. It's a chain: retrieve context, build a prompt, call a model, call a tool, validate, respond.
Without a trace ID connecting every step, debugging a bad response means guessing which of five separate log streams to check first. With one, a single query reconstructs the entire path an answer took—what was retrieved, what was sent to the model, what the model returned, what the tool did with it.
Distributed tracing wasn't optional for microservices. It isn't optional for multi-step AI workflows either.
Retrieval Quality, Not Just Model Output
A wrong answer built on irrelevant context is not a model problem. It's a retrieval problem the model faithfully reported.
Measuring retrieval separately—relevance scores, hit rate, how often results come back empty—tells you whether to fix the model prompt or fix the search index. Those are different teams, different fixes, and often a completely different root cause than the one that seemed obvious from the final output.
Tool-Call Success Rates
When a workflow calls external tools, a silent tool failure and a model reasoning failure can look identical from the outside: a wrong or incomplete final answer.
Logging tool-call success rates separately—including schema-validation failures, not just HTTP errors—stops teams from tuning a prompt to fix what was actually a broken tool integration.
User Corrections Are the Cheapest Signal You Have
Every time a user edits, rejects, or overrides an AI output, they are hand-labelling a failure for free.
Most systems throw this away. The correction happens in the UI and nothing downstream ever hears about it. Capturing it—even as a simple accept/edit/reject flag per response—turns real usage into a continuous, low-cost quality signal that no benchmark can replicate.
Sampling for What Metrics Miss
Automated metrics catch what they were built to catch. They miss tone drift, subtly outdated information, and answers that are technically correct but unhelpful.
A regular, small, random sample of real outputs reviewed by a human closes that gap. It doesn't need to be large to be useful—consistency over time matters more than volume in any single review.
Logging Without Leaking
Deep observability for AI systems often means logging prompts, retrieved context, and model outputs in full—which can mean logging exactly the sensitive data the system was trusted to handle carefully.
Treat this as a design constraint, not an afterthought:
- Redact or hash known-sensitive fields before they hit a log store
- Set retention limits on raw prompts and outputs
- Restrict who can query logs containing user content
- Prefer structured metadata over full text where the metadata is enough
Signals Worth Tracking
Pulling the threads above together, here's the minimum signal set for a workflow you actually intend to operate:
| Signal | Why it matters |
|---|---|
| Prompt & system-instruction version | So a quality shift can be traced to a specific edit |
| Model & configuration version | Providers change models under the same name without notice |
| Trace ID across the workflow | Connects retrieval, model calls and tool calls into one story |
| Token usage & latency per step | Separates a slow model call from a slow tool call |
| Retrieval relevance / hit rate | A wrong answer is often a retrieval problem wearing a model costume |
| Tool-call success rate | Silent tool failures look identical to model failures downstream |
| User corrections & overrides | The clearest, cheapest signal that an output was wrong |
| Sampled output-quality reviews | Automated metrics miss what a human glance catches immediately |
Detecting Gradual Degradation
The scenario earlier in this article—three weeks of slow drift—is the failure mode dashboards built around thresholds and alerts are worst at catching. Nothing ever crosses a red line; it just gets a little worse, repeatedly.
Catching it requires comparing against a trend, not a threshold:
- Track quality signals as a rolling week-over-week trend, not just a live value
- Alert on the rate of change, not only on absolute breaches
- Re-run a fixed set of golden test cases on every prompt or model change
- Review sampled outputs on a fixed cadence regardless of whether anything looks wrong
Trade-offs and Anti-Patterns
- Logging that the call succeeded without logging what was actually said
- Treating prompt edits as configuration, not as versioned, tracked changes
- Building dashboards for latency and cost while quality has no chart at all
- Discarding user corrections instead of feeding them back as a metric
- Over-logging raw sensitive content because redaction felt like it could wait
Final Thoughts
Internet operations teams learned that you can't fix what you can't see, and built tracing, structured logging and metrics to make the invisible visible. AI systems need the same investment, aimed at a harder target: not just whether a request completed, but whether it was any good.
A model that responds is observable.
A model whose reasoning you can reconstruct is operable.
A Practical Checklist
Before calling a workflow production-ready, work through these questions:
- 1Is every prompt and system instruction versioned, not just the code that calls it?
- 2Can you trace a single request across retrieval, model calls and tool calls with one ID?
- 3Do you log the exact model and configuration version used for each response?
- 4Do you measure retrieval quality separately from output quality?
- 5Is a sample of real outputs reviewed by a human on a regular cadence, not just after a complaint?
- 6Do user corrections and overrides feed back into a visible metric, not just a support ticket?
- 7Would a slow, gradual quality decline show up on a dashboard, or only in complaints?
- 8Does your logging redact or avoid capturing sensitive data by default?
About This Series
This article is part of the AI Infrastructure & Architecture series on Bourne Forge AI. It follows Designing AI Systems for Failure by covering how to see quality problems before they reach a user, ahead of the next article on context architecture—the system behind the prompt.
More from the Notes
Short technical notes and observations, written up as experiments produce something worth documenting.
Was this useful?
Published