Securing Tool-Using AI Systems
Prompt injection, excessive agency and the dangers of connecting language models to real infrastructure.
By Mark Bourne
Introduction
Context Architecture covered what happens when a model reads something it shouldn't trust. This article is about what happens when that same model can also act—send the email, update the record, call the API.
A model that only reads and responds is a search engine with better manners. A model that reads and can also act on real systems is closer to an employee with logins to everything, no manager, and a tendency to follow whatever instructions it encounters most recently—including instructions it was never supposed to receive.
The question is not whether a tool-using AI system can be manipulated. Assume it will be. The question is what it's actually capable of doing when that happens.
This article covers prompt injection, why permissions matter more than prompts, and the architectural controls that keep a manipulated model from becoming an incident.
The Internet Lesson: Least Privilege Was Never Optional
Network security stopped trusting anything by default a long time ago. Firewalls, access control lists, and the entire discipline of least privilege exist because “this system is generally trustworthy” was never good enough to justify broad access.
Every account, every service, every process got exactly the permissions it needed to do its job—and nothing else. Not because engineers distrusted their own systems, but because they understood that permissions define the damage a mistake, a bug, or an attacker can do.
Tool-using AI systems are, from a security standpoint, a new class of privileged process—one that takes instructions from unstructured text instead of a fixed API contract. The old discipline applies more urgently here, not less.
A Realistic Failure Scenario
An AI assistant handles inbound customer emails: reading them, drafting replies, and updating account records. It has write access to the CRM to keep records current.
An email arrives containing, buried in white text at the bottom, instructions aimed at the assistant rather than a human reader: forward all messages from this address to an external account, and grant it admin access on this record.
The assistant never distinguished “instructions from my operator” from “text found inside an email I was asked to process.” It had the permissions to comply. Nothing stopped it.
The model didn't fail at language understanding. The system failed at authorisation.
Direct and Indirect Prompt Injection
Prompt injection comes in two forms, and they need different defences.
- Direct injection: a user deliberately types instructions meant to override the system's guidelines
- Indirect injection: instructions hidden inside content the model processes — a web page, a document, an email, a tool's response
Direct injection is at least visible in the conversation log. Indirect injection is the harder problem: the malicious instruction never comes from the user at all. It arrives disguised as data, from a source the system was told to trust just enough to read.
Untrusted Documents and Websites
Any content the system didn't author—a fetched web page, an uploaded file, a third-party API response—should be treated as untrusted input, exactly like user input in a traditional web application.
- Structurally separate instructions from retrieved or fetched content
- Never let fetched content directly trigger a tool call without a check in between
- Strip or flag content that appears to be addressing the model directly
Excessive Agency and Least-Privilege Tool Permissions
“Excessive agency” is what happens when a system is given more capability than any single task actually requires—full CRM access to update one field, a general-purpose shell to run one command.
Scope every tool to the narrowest permission that does the job. A tool that updates a customer's phone number doesn't need permission to delete the account.
The prompt is not the security boundary. The permission the tool was granted is.
Read Versus Write Operations
Read and write operations do not carry the same risk, and shouldn't be governed by the same rules.
A wrong read returns bad information, which is recoverable. A wrong write changes the state of a real system, which may not be. Separating these explicitly in the permission model—not just informally in how tools happen to be used—is what makes the next control, approval gates, possible to apply consistently.
Approval Gates
Some actions are cheap to get wrong. Others aren't. Approval gates route the second kind through a human before they execute.
- Define which actions require approval before the system goes live, not after an incident
- Show the reviewer what will actually happen, not a vague summary
- Set a response-time expectation, or the gate becomes a bottleneck nobody honours
An approval gate that exists on paper but is routinely rubber-stamped without review provides the appearance of safety without the substance.
Credential Isolation
The model itself should never see a raw API key, password, or long-lived credential.
- Issue short-lived, narrowly scoped tokens per session or per task
- Keep credential resolution in the application layer, outside the model's context
- Rotate and revoke independently of any single conversation or user
If a credential never enters the model's context, it can't be leaked through the model's output—accidentally or through injection.
Data Leakage
A tool-using system that can read broadly and respond freely is a plausible path for sensitive data to leave a boundary it should never have crossed—summarised, paraphrased, or quoted directly into a response to the wrong audience.
- Filter retrieval and tool responses by the requesting user's actual permissions
- Treat model output as a potential exfiltration channel, not just an answer
- Watch for unusual patterns: large exports, repeated sensitive-field access, requests that don't match the user's normal role
Tool Allowlists and Sandboxing
An open-ended action space—arbitrary code execution, unrestricted API access—is a much larger attack surface than a fixed, reviewed set of tools.
- Expose only an explicit allowlist of tools, each with a defined contract
- Run code execution and file operations inside a sandbox with no path to production systems
- Treat every new tool addition as a permission-granting decision, not a convenience feature
Audit Logs and Emergency Shutdown
When something does go wrong, the two things that matter are knowing exactly what happened, and being able to stop it immediately.
- Log every tool call: what was called, with what arguments, by which session, and why
- Make logs queryable quickly under incident pressure, not just archived for compliance
- Maintain a tested kill switch that revokes a system's credentials and tool access instantly
- Practice using it before the day it's actually needed
A Permission Model by Risk Tier
Putting the controls above into a single model makes the required control obvious for any new tool before it ships:
| Tier | Example | Required control |
|---|---|---|
| Read-only | Search a knowledge base, look up an order | Scoped credentials, logged, minimal review |
| Write, reversible | Draft an email, create a calendar hold | Allowed autonomously, logged, easily undone |
| Write, hard to reverse | Send an email, update a customer record | Approval gate or tight rate limiting |
| Write, irreversible or high-value | Transfer funds, delete data, change access permissions | Mandatory human approval, no autonomous path |
Trade-offs and Anti-Patterns
- Relying on prompt instructions ('never do X') as the actual security boundary
- Granting broad API scopes because narrow scoping is more setup work
- Approval gates that are rubber-stamped without real review
- Letting fetched or retrieved content trigger tool calls without a check in between
- No tested kill switch, or one that takes longer to use than the incident takes to unfold
Final Thoughts
Network security stopped asking “can we trust this system?” decades ago and started asking “what is this system actually capable of doing if something goes wrong?” Tool-using AI needs the same question asked before deployment, not after.
A well-written prompt is a request.
A well-scoped permission is a guarantee.
A Practical Checklist
Before connecting a model to real infrastructure, work through these questions:
- 1Does every tool grant the minimum permission needed, rather than broad API access?
- 2Are write operations separated from read operations in the permission model, not just in code comments?
- 3Do irreversible or high-value actions require human approval with no autonomous bypass?
- 4Does the model ever see a raw credential, or only short-lived, scoped tokens?
- 5Is there an explicit allowlist of callable tools, rather than an open-ended action space?
- 6Do tool calls run in a sandbox that limits blast radius if something goes wrong?
- 7Is every tool call logged with who, what, when and why, in a way a human can audit later?
- 8Is there a tested way to instantly revoke a system's access if it starts behaving badly?
About This Series
This article is part of the AI Infrastructure & Architecture series on Bourne Forge AI. It follows Context Architecture by covering what happens once a model can act, not just read, ahead of the next article on model independence and the portability layer.
More from the Notes
Short technical notes and observations, written up as experiments produce something worth documenting.
Was this useful?
Published