What Changes When AI Acts: Governing Autonomous Agents

A client called me last month about an agent their operations team had quietly turned on. It was meant to draft replies to supplier emails. Within a week it was sending them, and on day nine it agreed to a price change with a vendor because the prompt scaffolding had drifted and nobody had wired a hard stop. The signed PDF was already in the vendor’s archive before anyone in finance had read the thread. Nothing illegal happened. Nothing dramatic. Just a small contract that the company now had to honour, made by software that was supposed to be drafting, not deciding.

I keep walking into versions of this scene. The mood is never panic. It is the quiet realisation that the system crossed a line nobody had drawn, because the line had never been drawn. That is the whole story of what changes when AI acts.

Answer versus act is the whole risk shift

For two years the dominant AI failure mode was a wrong sentence. The model said something inaccurate, a human read it, the human corrected or discarded it. Embarrassing, sometimes expensive, rarely catastrophic. The human was the load-bearing wall between the model and the world.

When the model acts, that wall comes out. A wrong sentence becomes a wrong action: a booked meeting, a moved budget line, a refund issued, a record updated, a message sent on the company’s name. The error does not stop at the model’s output; it lands in a downstream system and starts having consequences before anyone reads it.

This is not a difference in degree. It is a category change. An assistant that hallucinates is a quality problem. An agent that acts on the same hallucination is a liability event. The legal, financial, and reputational consequences scale with the action, not with the sentence.

The four new things agents introduce

When I sit with operations teams trying to put words on what feels different, four things come up every time. They are not exotic; they are just the parts traditional QA was never built for.

Autonomous action. The agent does not propose, it executes. A traditional system waits for a human click between intent and effect. An agent treats the human click as optional, or removes it entirely. The control surface that used to live in the user interface now has to live inside the agent’s authorisation rules.

Real-time speed. A human reviewer can read a hundred drafted emails a day. An agent can send ten thousand. When the action runs at the speed of the data flowing past it, any control that depends on human review-before-send is mathematically broken. Either the human becomes a rubber stamp, or the queue piles up and the business pulls the human out of the loop to make the system useful.

Downstream propagation. A wrong sentence dies on the page. A wrong action writes to a CRM, fires a webhook, updates a ledger, triggers a workflow. By the time the error is noticed, three other systems have already trusted it. Rolling back takes longer than it took to commit the mistake, and some downstream consequences (a sent email, a published post, a paid invoice) cannot be rolled back at all.

Composition with other agents. This is the one that bends the curve. Agents now call other agents, often across vendor boundaries. One agent’s output is another agent’s input, and neither of them is a human. The error compounds along the chain, and attribution becomes a forensic exercise: which agent made the call that broke the contract? The pattern looks a lot like algorithmic trading in 2010, where independent rule-following systems combined into market events no single party had authored. We are quietly rebuilding that topology for back-office work.

Why traditional QA misses all four

Traditional software QA tests the system before it ships and monitors the system after it ships. Both halves assume the system does roughly the same thing every time. An agent does not. It composes a new sequence of actions for each input, drawing on tools, memory, and context that change between calls. There is no fixed test surface, because the surface itself is generated at runtime.

The QA mindset that catches a typo in a form field cannot catch an agent that decides to use a tool it has never used before, in a combination nobody anticipated, because the prompt context made that combination look reasonable in the moment. That is not a bug in the agent. It is the design.

So the controls that matter shift. They stop being pre-shipment tests and become runtime guardrails plus after-the-fact reconstruction. You constrain what the agent can do (authorisation), you record everything it did (audit), you give yourself a way to stop it cold (hard stop), you keep a human on consequential calls (checkpoint), and you tell the agent when its job is over (termination condition). None of these are exotic. They are the same controls we used for any system that could act on its own, applied on purpose this time.

The minimal control set that closes most of the gap

Authorisation. A whitelist of tools, scopes, and data the agent may touch, enforced at the tool layer not the prompt layer. Prompts are wishes; the authorisation table is law.

Audit. Every call, every input, every output, every tool invocation, retained. Without this you cannot reconstruct what the agent did, which means you cannot defend it to a regulator or fix it for next time.

Hard stop. A kill switch that any on-call human can pull without a meeting. The test of a hard stop is not whether it exists; it is whether the on-call engineer at 02:00 knows the exact command and trusts it will not be second-guessed in the post-mortem.

Human checkpoint. A bright line above which the agent must request approval. Draw the line honestly: too high and the human rubber-stamps everything, too low and the system stops being useful. The art is in the calibration, not the existence of the checkpoint.

Termination condition. An explicit answer to “when is this agent done.” Without it, agents loop, drift, or quietly keep working on a goal nobody remembers asking for. The most common failure mode I see in production agents is not a wrong action; it is an action taken long after the original task ended, because the agent never received a signal that the job was over. We cover the wiring of these guardrails in Authorisation and Guardrails for AI Agents and the recording side in Audit and Observability for AI Agents.

Why GRC needs a fourth pillar

The classical GRC operating model watches three actors: people, processes, and systems. Agents do not fit cleanly into any of the three. They are not people (no employment relationship, no judgement, no accountability), they are not processes (they generate the process at runtime), and they are not systems in the legacy sense (they decide rather than execute fixed logic). Treating them as one of the three under-protects against the others.

The cleanest framing I have seen is to add a fourth pillar: autonomous actors, governed alongside the existing three with their own controls, owners, and review cadence. The NIST AI Risk Management Framework‘s generative AI profile gestures in this direction, and agent-safety research from Anthropic and the Allen Institute for AI is converging on roughly the same control list (authorisation, audit, checkpoint, termination) from the technical side. The discipline lines up. The operating model has to catch up. We work through the operating-model details in Building an AI Governance Framework: GRC for Autonomous Systems.

This is the part to take seriously. An assistant that says something wrong is embarrassing. An agent that does something wrong is a liability event. The risk arithmetic changed the day the model started touching the keyboard. Govern the action, not the answer.