AI Cost and Performance Engineering
Managing tokens, latency, caching and capacity without weakening the user experience.
By Mark Bourne
Introduction
Human-in-the-Loop as an Architectural Component treated review capacity as an operational resource. This article treats compute cost the same way.
Defining Reliability for AI Systems introduced “cost per success” as a service-level indicator, and Model Independence noted that routing becomes a real option once providers sit behind a common interface. This article is where those threads get assembled into an actual cost and performance discipline.
The most expensive model, called for every request regardless of how simple, isn't generosity toward quality. It's an unmanaged cost curve waiting to become a budget conversation nobody wanted to have.
This article covers where AI spend actually goes, the levers that bring it under control, and how to pull them without users noticing anything except, occasionally, that the system got a little faster.
The Internet Lesson: Manage Demand, Don't Just Buy Bandwidth
Networks never solved congestion by assuming infinite capacity. TCP congestion control, queuing disciplines, and traffic prioritisation exist because bandwidth is finite and demand is elastic—it expands to fill whatever capacity is available.
The lesson wasn't “buy bigger pipes forever.” It was: measure demand, prioritise what actually needs the expensive path, and degrade gracefully when load exceeds capacity rather than letting everything compete for the same resource equally.
AI compute is the new bandwidth question, and it responds to the same discipline.
A Realistic Failure Scenario
A team ships a chatbot that routes every request—from “what are your hours” to genuinely complex troubleshooting—to the most capable model available, with no caching layer.
Usage grows. The bill grows faster, because it scales linearly with a traffic pattern nobody modelled in advance. When finance flags it, the fix arrives under pressure: an emergency downgrade to a cheaper model, applied uniformly, overnight.
Quality drops on exactly the complex requests that needed the capable model, because the downgrade wasn't designed—it was a blunt instrument applied in a hurry. Users notice within days.
The cost problem was real. The damage came from having no tiered plan ready when it surfaced.
Cost Per Completed Workflow, Not Per API Call
A single API call is the wrong unit of cost measurement. A workflow that fails partway through and retries from scratch can cost several multiples of one that succeeds cleanly on the first attempt.
Measuring cost per completed workflow—including retries, fallback calls, and wasted partial attempts—reveals where the real spend is happening, which is often not where the raw per-call pricing suggests.
Cost Per Successful Answer
Completion isn't success. A workflow that finishes but produces a wrong or unhelpful answer, requiring a human to redo the work, cost more than the API bill shows.
Cost per successful, correct outcome is the number that actually matters for a build-vs-buy or model-choice decision—a cheaper model with a lower success rate is not automatically the cheaper option once rework is priced in.
Token Budgets
Without an explicit budget, context tends to grow until something breaks: a document gets fully included instead of summarised, a conversation history gets replayed in full every turn.
- Set a token budget per step, not just a global limit
- Decide in advance what gets trimmed first when a step approaches its budget
- Alert when a workflow's typical token usage drifts upward over time
Model Routing by Task Difficulty
Not every request needs the most capable available model. A classification task, a simple lookup, or a templated response often performs identically on a smaller, cheaper model.
- Classify incoming requests by estimated difficulty before routing
- Reserve the most capable model for requests that demonstrably need it
- Monitor whether cheaper-tier routing is quietly degrading quality, and adjust the split
This is the practical payoff of the portability layer described in Model Independence: routing by difficulty is a configuration change, not a rewrite.
Two Kinds of Caching
Caching remains one of the highest-leverage cost tools available, and AI workflows benefit from two distinct forms of it.
- Exact-response caching: identical requests return a stored response, at near-zero marginal cost
- Semantic caching: requests that are meaningfully equivalent, though not identical in wording, share a cached response
Exact caching is safe and simple, but catches less traffic than it could. Semantic caching catches more, at the cost of occasionally matching a near-miss that wasn't really the same question—a trade-off worth tuning deliberately, not adopting blindly.
Batch Processing
Not every workflow needs a real-time response. Overnight report generation, bulk document classification, and scheduled summarisation can run as batches, which many providers price significantly below real-time inference.
The cost saving is real, but it only applies where latency genuinely doesn't matter. Routing a user-facing, time-sensitive request into a batch queue trades cost for a reliability failure of its own.
Smaller-Model Fallbacks, Designed in Advance
Designing AI Systems for Failure covered reduced-capability modes for outages. The same pattern applies to cost pressure: a tested, deliberate fallback to a smaller model beats an emergency downgrade applied under deadline pressure to the entire system at once.
Design it, test its quality impact, and know in advance which workflows can absorb it and which can't—before the budget conversation forces the question.
Context Compression
A large context window makes it easy to include everything that might be relevant. Every token included is a token paid for, whether or not it changes the answer.
- Summarise long conversation history instead of replaying it in full each turn
- Retrieve and include only the passages that scored as relevant, not the whole source document
- Periodically compress older context into a condensed summary rather than letting it accumulate
Latency Budgets Are Cost Decisions Too
Defining Reliability for AI Systems covered latency as a reliability target. It's also a cost lever: a faster model tier, aggressive caching, or parallelised retrieval all trade engineering effort for latency—and latency reduction often has a cost, not just a benefit.
Set the latency budget based on what the workflow actually needs, not the fastest achievable number regardless of cost.
The Cost-Quality-Latency Triangle
Every lever in this article trades against the other two corners of the same triangle: cost, quality, and latency. Improving one without a deliberate trade-off usually means spending on another.
| Lever | What it trades |
|---|---|
| Route by task difficulty | Engineering effort for lower average cost per request |
| Exact-response caching | Storage and cache-invalidation care for near-zero repeat cost |
| Semantic caching | Some risk of a near-miss for broader cache hit rates |
| Batch processing | Latency (delayed results) for significantly lower per-unit cost |
| Context compression | A little upfront summarisation cost for a smaller ongoing token bill |
| Smaller-model fallback | Peak-condition quality for sustainable cost under load |
There is no lever here that's free. The goal is choosing which trade-off suits each workflow, rather than applying one setting everywhere.
Environmental Considerations
Every token processed has an energy cost. It's a small one per request, and a meaningful one at the volume most production systems eventually reach.
The good news is that most of the levers in this article—caching, routing by difficulty, context compression, avoiding wasted retries—reduce both cost and energy use at the same time. Cost discipline and environmental impact point in the same direction more often than either gets credit for.
Trade-offs and Anti-Patterns
- Routing every request to the most capable model regardless of task difficulty
- No caching layer at all, exact or semantic
- Measuring cost per API call instead of per completed, successful workflow
- An emergency, undesigned downgrade as the only response to a cost spike
- Batching latency-sensitive, user-facing requests to save money
- Including full documents in context when a relevant excerpt would do
Final Thoughts
Network engineers learned that managing demand beats endlessly buying capacity, and built an entire discipline—queuing, prioritisation, congestion control—around that idea. AI cost engineering is the same discipline, aimed at tokens instead of packets.
Unmanaged cost isn't a sign of ambition.
It's a sign the routing, caching and budgets were never designed.
A Practical Checklist
Before scaling a workflow's traffic, work through these questions:
- 1Do you measure cost per completed, correct workflow, not just cost per API call?
- 2Is there a token budget per step, with a defined response when a step exceeds it?
- 3Are simple requests routed to a cheaper model, or does everything hit the most capable one?
- 4Do you have both exact-response and semantic caching where either would apply?
- 5Can non-urgent workloads run in batch instead of real time?
- 6Is context trimmed or compressed before it's sent, or is every token from every source included by default?
- 7Is there a tested, deliberate fallback to a smaller model under cost or capacity pressure?
- 8Would a 10x traffic spike degrade gracefully, or blow through budget before anyone noticed?
About This Series
This article is part of the AI Infrastructure & Architecture series on Bourne Forge AI. It follows Human-in-the-Loop as an Architectural Component by turning cost and performance into a designed set of trade-offs, ahead of the next article on change management for AI systems.
More from the Notes
Short technical notes and observations, written up as experiments produce something worth documenting.
Was this useful?
Published