Context Architecture: The System Behind the Prompt
Why information provenance, freshness and retrieval matter more than clever prompting.
By Mark Bourne
Introduction
Internet Architecture Lessons for AI Systems made a claim worth repeating on its own: context is the new bandwidth.
Most effort spent “improving” an AI system goes into the prompt—better phrasing, more examples, sharper instructions. Most of the actual quality ceiling is set somewhere else entirely: what information the model was given before it ever started reasoning.
The best model in the world, handed the wrong document, will confidently answer from the wrong document.
No amount of prompt engineering fixes bad context. It just produces a more articulate version of the wrong answer.
This article is about the system that decides what a model sees: where information comes from, how it's judged trustworthy, how it's kept current, and how conflicts and gaps get handled before they become confident nonsense.
The Internet Lesson: Bandwidth Wasn't the Bottleneck for Long
Early Internet performance work obsessed over bandwidth—bigger pipes, faster links.
It didn't take long to discover that raw bandwidth stopped being the limiting factor once it was merely adequate. What mattered after that was routing the right packets to the right place, caching what was requested often, and not sending garbage down the pipe in the first place.
AI is following the same arc. Context window sizes keep growing, and larger windows solve less than the marketing suggests, because the problem was never how much context could fit. It was whether the right context made it in.
A bigger context window filled with irrelevant or wrong information is not progress.
A Realistic Failure Scenario
A policy assistant answers HR questions from a document index. Two versions of the leave policy exist in the index: the current one, and a superseded draft that was never removed after the rewrite.
The retrieval system has no concept of “superseded.” Both documents look equally valid to a similarity search. On a bad day, the draft ranks higher because its phrasing happens to match the question more closely.
The assistant answers confidently, citing a document that looks official, quoting a policy that hasn't applied for six months.
The model reasoned correctly. The retrieval system handed it a landmine.
Context Windows Are Not Persistent Knowledge
It's tempting to treat a large context window as a substitute for a real knowledge system. It isn't.
A context window is working memory for a single request. It has no concept of what was true yesterday, no built-in authority ranking, and no memory once the request ends. Persistent knowledge—what's current, what's authoritative, what's been retired—has to be maintained outside the model, in the retrieval system that feeds it.
Stuffing more into the window doesn't create a knowledge system. It creates a bigger haystack.
Retrieval-Augmented Generation, Honestly Assessed
RAG solves a real problem: giving a model access to information beyond its training data, without retraining it. It does not, by itself, solve the harder problem—making sure what gets retrieved is correct, current and relevant.
A RAG pipeline with careless chunking, no freshness handling and no authority ranking will retrieve confidently and be wrong just as confidently. Retrieval is not a correctness guarantee. It's a plumbing layer, and plumbing can carry clean water or contaminated water equally well.
Source Authority and Provenance
Not all information deserves equal trust, and most retrieval systems don't model that distinction at all—a signed policy document and an old Slack thread can rank identically if their embeddings happen to be similar.
| Question | Why it matters |
|---|---|
| Where did this come from? | Unattributed context can't be trusted or debugged |
| Is it authoritative? | A wiki page and a signed policy document aren't equally reliable |
| Is it current? | Correct-when-written isn't the same as correct-now |
| Who is allowed to see it? | Retrieval can leak access-controlled data if permissions aren't enforced at query time |
| What happens if it conflicts with another source? | Silent conflict resolution hides the disagreement instead of surfacing it |
None of these questions are exotic. They're the same questions a careful editor asks before citing a source—just rarely asked of a retrieval index.
Freshness, Expiration and Conflicting Sources
Documents go stale. Policies get superseded. Prices change. A retrieval index that never expires anything accumulates landmines exactly like the one in the scenario above.
- Tag documents with effective dates and expiry, not just creation dates
- Remove or clearly demote superseded versions instead of leaving them retrievable
- When sources conflict, surface the conflict rather than silently picking one
- Prefer the most recent authoritative source by default, and say so in the answer
An assistant that says “I found two conflicting policies and used the more recent one” is more trustworthy than one that picks silently and sounds equally confident either way.
Chunking and Ranking
How a document gets split, and how results get ranked, quietly shapes every answer built on top of it.
Chunking around arbitrary character counts routinely:
- Splits a clause from the condition that governs it
- Separates a table from its caption
- Cuts a numbered list in half across two chunks
Chunking around meaning—sections, clauses, logical units—keeps context intact. Ranking by semantic similarity alone ignores authority and freshness entirely; a ranking function that folds in source trust and recency, not just embedding distance, retrieves noticeably better context for the same underlying documents.
Access-Controlled Retrieval
A retrieval index built from documents with different access levels needs to enforce those access levels at query time, not just at document-ingestion time.
If the index doesn't know that a user shouldn't see a document, retrieval can quietly surface it inside an answer—a data leak that never shows up in an access log, because no one directly requested the document. They asked a question, and the answer happened to contain it.
Context Poisoning
If untrusted content—a scraped web page, an uploaded file, a public wiki edit—can enter the retrieval index, it can also carry instructions aimed at the model rather than the reader: hidden text telling the assistant to ignore its guidelines or leak data.
Treat any content the system didn't author as untrusted input:
- Separate instructions from retrieved content structurally, not just by convention
- Sanitise or strip content that looks like it's addressing the model directly
- Restrict which sources are eligible for retrieval into privileged workflows
Detecting Missing Context
The most dangerous retrieval failure isn't returning the wrong document. It's returning nothing useful and having the model fill the gap with a plausible guess.
A system that checks retrieval confidence and explicitly says “I don't have enough information to answer that” is more useful than one that always sounds sure.
Missing context is recoverable.
A confident answer built on nothing usually isn't caught until it's acted on.
Show Citations to Users
The simplest trust-building feature in a retrieval-backed system is also the most underused: showing which source an answer came from.
Citations let a user catch a stale or wrong source in seconds, without needing the system to be perfect. They turn “trust the AI” into “verify the AI, quickly”—a much easier bar to clear, and a much safer one to rely on.
Trade-offs and Anti-Patterns
- Treating a larger context window as a substitute for retrieval discipline
- Letting superseded documents stay retrievable indefinitely
- Chunking purely by character count instead of by meaning
- Ranking by similarity alone, ignoring authority and freshness
- Enforcing access control on ingestion but not on retrieval
- Hiding missing context behind a confident-sounding guess
Final Thoughts
Internet architects learned that moving data fast is easy, and moving the right data, from a trustworthy source, at the right time, is the actual engineering problem. Context architecture is that same problem, wearing a different name.
The model reasons over what it's given.
Everything upstream of that decides whether the reasoning was worth doing.
A Practical Checklist
Before trusting a retrieval-backed workflow, work through these questions:
- 1Does every piece of retrieved context carry a source and a timestamp?
- 2Is there a defined authority ranking for when two sources disagree?
- 3Are stale documents expired or flagged, rather than retrievable forever?
- 4Is retrieval filtered by the requesting user's actual permissions, not just the index as a whole?
- 5Are chunks sized around meaning (sections, clauses) rather than arbitrary character counts?
- 6Does ranking account for authority and freshness, not only semantic similarity?
- 7Can the system detect and say 'I don't have enough context' instead of guessing?
- 8Are citations shown to users so a claim can be checked against its source?
About This Series
This article is part of the AI Infrastructure & Architecture series on Bourne Forge AI. It follows Observability for Non-Deterministic Systems by examining the information supply chain behind every prompt, ahead of the next article on securing tool-using AI systems.
More from the Notes
Short technical notes and observations, written up as experiments produce something worth documenting.
Was this useful?
Published