Threat modelling
AI threat modelling: give AI as much autonomy as the risk allows
I've started looking at AI automation slightly differently. Before asking "what can we automate?", we should also be asking "what could go wrong if we automate this?"
It's basically threat modelling, applied to AI. Let's call it AI threat modelling.
The questions I ask about every agent
If an agent is going to read customer data, update systems, send messages or trigger actions, I want to understand:
- What does it have access to: which data, which systems, which accounts?
- What could it do if it made the wrong decision?
- What happens if someone manipulates the input it's reading?
- Which actions need human approval?
- Can we see exactly what it did afterwards, and why?
None of those questions are exotic. They're the same questions a good security architect asks about any system. The difference with AI is that the component making decisions can be influenced by the content it reads, so the answers matter even more.
Then design the workflow around the risk
Once you know the answers, you can draw sensible lines. Maybe the AI can read an enquiry, qualify it, update the CRM and draft a response. That's useful, low-regret work, and a mistake is easy to spot and correct.
But should it automatically issue a refund, delete customer data or approve a large payment? Ehh… something tells me probably not.
Those are high-impact, hard-to-reverse actions. They're exactly where you want least-privilege permissions, an approval step at the action boundary and an audit trail that shows who approved what.
A simple way to think about it
- List every action the agent can take.
- For each one, ask how bad it would be if the action were wrong, manipulated or repeated a thousand times.
- Let low-impact, reversible actions run autonomously, with logging.
- Put human approval, tighter permissions or extra validation in front of anything high-impact or irreversible.
- Revisit the list whenever you add a tool, a data source or a new workflow.
The goal isn't to automate as much as possible. It's to give AI as much autonomy as the risk allows. That's the difference between AI that looks impressive in a demo and AI you can actually trust to run part of a business.