65% of companies running AI agents in production have already had an incident caused by one. That is not a statistic about the technology. It is a statistic about ownership.
The pattern behind the incident
Almost no agent incident comes from the model “getting it wrong”. It comes from the agent having permission to do something nobody thought to limit:
- A support agent with write access to the production database, not just read.
- A reconciliation agent that retries a financial action without idempotency.
- An agent that chains three tools, where the third was never audited because “it was only a fallback”.
In all three cases the model did exactly what it was asked. The problem is the perimeter the team drew around it.
Demo versus production
An agent demo is judged on whether it completes the task. An agent in production is judged on what happens when something fails halfway through a chain of actions. That is the question almost nobody asks before the first deploy:
// The wrong question
did_the_agent_complete_the_task === true;
// The right question
can_this_agent_take_an_irreversible_action_without_a_human_or_a_hard_check_in_the_loop ===
false;What changes when the design is serious
Designing agentic systems for real environments means treating every tool the agent can invoke as a risk surface, not as a feature. In practice:
- Explicit scopes per tool, not one token with blanket access.
- Irreversible actions — send, pay, delete, publish — always behind a confirmation or a hard rule.
- Logs that let you reconstruct why the agent decided what it did, not just what it returned.
The agent is not the risk. The risk is treating it as deterministic when it is not.