Picture the incident review. An AI agent processing supplier invoices approved a payment it should not have. Not a fortune, but enough to get noticed. The finance director asks a simple question: who is responsible?
The vendor says the model behaved within its documented limits. The engineering team says they built what was specified. The process owner says they assumed IT had it covered. The agent, of course, says nothing.
This is the AI accountability gap, and as agents take on more real work, it is getting wider.
The pressure is coming from an unexpected place
Most people assume regulation will close the gap. Gartner thinks otherwise. In its strategic predictions for 2027 and beyond, it forecasts that by 2030, insurers, not regulators, will drive AI governance, through strict liability underwriting standards for AI. It also predicts that by 2030, 80% of Global 500 companies will contractually make their CIO or chief AI officer an "evidence custodian", accountable for AI actions (Gartner).
That rings true. Regulation moves slowly and varies by country; even the EU has pushed back its high-risk AI deadlines. Insurers move as soon as claims start landing. When an insurer asks, at renewal, "show us how your AI agents are controlled and what evidence you keep", governance stops being a policy document and becomes a condition of cover.
Accountability is a design choice
The good news is that accountability for AI is not mysterious. It works much like accountability for any other powerful system or team. It needs a named owner, clear boundaries, a record of what happened and a plan for when things go wrong.
One named owner per agent. Not a committee, not "the AI team". A specific person who owns the outcome the agent is meant to deliver, and who decides whether it keeps running. Usually that is the business owner of the process, supported by the technical team that runs it.
Written boundaries. What the agent may do, what it may never do, and what it must escalate. Which systems it can touch, with which permissions, up to what value. If the boundaries are not written down, they do not exist.
Evidence by default. Every decision and action logged: what the agent saw, what it decided, which tools it called and what happened. This is what Gartner means by evidence custody. When something goes wrong, you need to reconstruct events in minutes, not weeks. When an insurer or auditor asks, you need to show it.
A plan for the bad day. How to pause an agent quickly, how to reverse its actions where possible, who to tell and how to learn from it. Agent incidents should go through the same blameless review process as any other production incident.
The question to ask at your next board meeting
Boards do not need to understand how a language model works. They do need to know, for each AI agent acting on the organisation's behalf, who owns it, what it is allowed to do and what evidence exists of what it has done.
If that question gets a confident answer, you are ahead of most. If it gets a long silence, that is your governance roadmap for the next quarter.
Why this is good for adoption
It is tempting to see accountability as a brake on AI. In practice, it is the opposite. Teams are far more willing to let agents take on meaningful work when they know the boundaries are clear and the evidence will be there if something goes wrong. Clear ownership turns AI from a risk that someone might be blamed for into a capability someone is proud to run.
The organisations that build this now, before an insurer or regulator forces it, will scale agents faster and with fewer nasty surprises. For the technical side of the same question, our note on securing AI agents that can act covers permissions, isolation and logging in detail.
If you want help designing an accountability model for the agents you are running or planning, our Consulting team can help you put it in place.
Frequently asked questions
Who is responsible when an AI agent makes a mistake?
Inside an organisation, responsibility should sit with a named owner for each agent, usually the business owner of the process it supports, backed by clear boundaries and a complete record of the agent's actions.
Will insurers drive AI governance?
Gartner predicts that by 2030 insurers, rather than regulators, will drive AI governance through strict liability underwriting standards for AI.
What is an AI evidence custodian?
It is Gartner's term for an executive, such as a CIO or chief AI officer, who is contractually accountable for keeping evidence of AI actions and decisions.