AI agent governance

AI agent governance is governance applied to AI systems that act: deciding what agents may do and under whose authority, evaluating what they actually do, and keeping evidence of both. It extends AI governance from models and their outputs to actions and their effects.

A model that writes a wrong answer produces a bad paragraph. An agent that takes a wrong action changes a real system: it deletes a record, sends a message, moves money, opens access. Governing agents therefore means more than reviewing what a model says. It means being able to answer, for every agent and every run, six questions:

1. What was the agent authorized to do? The systems, tools and data it could reach, and who granted them.
2. What was it asked to do? The objective and scope of this particular task.
3. What did it attempt? The actions it actually took, in order.
4. What did that achieve? Whether the requested outcome exists, verified against the systems concerned.
5. What did it cost? In tokens, money, time, retries and human attention.
6. Who answers for it? The people accountable for the agent, its authority and its policies.

Answering all six establishes the evidence and accountability that governance depends on. It is not governance on its own: someone still has to oversee agents, intervene when an answer is wrong, and correct what caused it. An organization that can answer only the first is managing permissions.

Why AI agents need their own governance

AI governance already covers whole systems: their purpose, data, evaluation, deployment, monitoring and oversight. Frameworks such as the NIST AI Risk Management Framework and ISO/IEC 42001 give it an organizational shape. Agents keep all of those obligations and add new ones.

Four properties drive the difference.

• Autonomy. An AI agent chooses its own sequence of steps. What needs governing is a run that nobody wrote in advance.
• Tool-mediated execution. The output is an operation on a real system. OWASP calls the resulting risk excessive agency: more functionality, permission or autonomy than the task requires.
• Delegation and scale. Agents call other agents, use third-party tools and run in many places at once, so no single system’s log shows the whole picture.
• Persistence. Agents increasingly carry memory across sessions and run for hours or days, so what they learned or were told yesterday shapes what they do today.

The stakes are practical as well as ethical. Gartner predicts that over 40 percent of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls. Those are three different governance failures, and they map onto the three responsibilities below.

Agent governance is related to, and distinct from, AI agent security, which protects agents and the systems they reach against misuse and attack. A well-secured agent can still do the wrong job.

Three responsibilities: compliance, completion, efficiency

It is tempting to treat governance as the question of whether an agent stayed within the rules. That is one responsibility of three.

Compliance. Did the agent act within its authority, its declared scope and the applicable policy? This is the most familiar responsibility and the one most tooling addresses.

Completion. An agent can stay inside its permissions while failing to finish, pursuing something irrelevant, or reporting success without the requested outcome existing. Four things are easy to confuse:

Only the last two say the job was done. The original objective is what makes them measurable, which is why it has to stay in view for the whole run.

Efficiency. What did successful completion cost? Cost per successful completion is more informative than cost per action or per model call, because it counts the retries, failures and human intervention a per-call figure hides. Efficiency is not the fewest steps: re-running tests or asking a person before an irreversible change adds work and is often justified. The question is whether the work was proportionate to the objective, by limits the organization has set.

The AI agent governance stack

No single control governs an agent. Governance is a stack of controls that act at different moments, each answering some of the six questions. The parts below are organized by when they act.

Before an agent runs

• Inventory and ownership. Every agent in use, its purpose, its owner and what it can reach, including agents nobody formally approved. OpenAI’s practices for governing agentic systems suggest giving each agent instance a unique identifier, “similar to business registrations”.
• Agent identity and authority. Agents need their own identities rather than borrowed human or shared service credentials, with permissions granted on the principle of least privilege. Identity platforms now record a human sponsor for each agent, so that authority traces back to a person.
• Risk assessment. Classify each agent by how much autonomy it has and what its actions can affect. Singapore’s framework for agentic AI starts here: assess and bound the risks upfront, by setting limits on agent autonomy, tools and data access.
• Policy and completion criteria. Write down what the agent is expected to do and not do, which operations are high-consequence, what counts as done, and what resources a task may use. Keeping these expectations as reviewable, versioned configuration, a governance profile, makes them consistent across agents and auditable later.
• Evaluation. Test the agent on representative tasks, including whether it completes them, before it acts on real systems.

While an agent runs

• Gateways and authorization. Mediate requests at the boundaries they control, and enforce what the agent may reach.
• Guardrails. Check inputs, outputs and tool calls against constraints defined in advance.
• Runtime governance. Evaluate each attempted action against the agent’s declared intent, scope and policy, and keep the result as governance evidence. This is what connects the authority decided beforehand with the review that follows.
• Human oversight at chosen points, especially before high-consequence actions.
• Observability, tracing and monitoring. Insight into how a run unfolded and what it cost, and alerts when measures move.
• Interruptibility. A way to stop an agent. OpenAI’s practices call the ability to turn an agent off “a critical backstop”, and the EU AI Act requires that the people overseeing a high-risk system be able, as appropriate and proportionate, to interrupt it through a “stop” button or a similar procedure.

Autonomy needs room inside these controls. An agent has to respond to what it learns, and a change of approach is not automatically drift. Governance does not settle that tension by formula: it keeps the objective in view, so that an adaptation can be recognized as one and a departure as a departure.

Across sessions

Agents increasingly remember. They carry memory between sessions, compress long contexts to keep working, and run for hours. Governance has to persist through all of that:

• the objective and the rules an agent runs under have to survive context compaction and session boundaries;
• what an agent writes to memory is a decision with consequences for every later session;
• behavior is best compared across sessions, not only within one, because slow drift is invisible inside any single run.

This is the least mature part of the stack, and one of the most important as agents become long-lived.

After an agent runs

• Verification. Confirm outcomes against the systems acted on, not the agent’s report.
• Review and incident response. Investigate departures and failures, roll back where possible, and reconstruct what happened from the record.
• Continuous improvement. Change policies, scopes, limits and approval points based on what review finds, and measure whether the changes worked: departure rates, verified completion rates, cost per successful completion, approval rates.

Closing the loop

Governance becomes continuous when evidence from one run shapes the next.

1. Runtime evidence records what each agent was asked to do, what it attempted and how each attempt related to its task and policy.
2. Evaluation turns that evidence into tests. A departure, a missed completion or a costly retry becomes a case the agent, or its next version, is evaluated against before it runs again.
3. Human review reads the same evidence to judge what the findings mean: a reasonable adaptation, a policy that was too narrow, or a real failure.
4. Policy refinement changes scopes, limits, designations and approval points in response.
5. The next execution runs under the refined policy, and its evidence starts the loop again.

One boundary holds throughout. Changes to policy and authority are made by the people accountable for them, through the same approval as the original grant. An agent can surface evidence that its scope is too narrow, and can even propose a change, but it does not widen its own permissions or rewrite the rules it runs under. Each change should be versioned and recorded, so that every later finding can be traced to the policy that produced it.

Scaling governance to autonomy and consequence

Governing every agent action with the same scrutiny fails in two directions. Too little, and consequential actions go unnoticed. Too much, and people asked to approve hundreds of routine actions learn to approve without reading, which is oversight in name only.

The answer is proportionality. Researchers have proposed levels of agent autonomy defined by the role a person plays, from operator through collaborator, consultant and approver to observer. The more autonomy an agent has, the more governance shifts from approving steps to designing limits, evidence and review. Within any level, the operations that are costly or hard to reverse deserve the most attention: designate them explicitly, put human approval there, and let routine, reversible work flow with a record. Human attention is the scarcest resource in agent governance, and it should be spent where judgment changes outcomes.

Who is responsible

Technology does not hold accountability. People and organizations do, and agent governance has to name them.

• Who deploys the agent, and approved it for this use.
• Who establishes its authority: the permissions and the systems in reach.
• Who owns its policies, and is entitled to change them.
• Who is accountable for its behavior when something goes wrong.

Singapore’s framework puts it plainly: humans are ultimately accountable. The organization also sets the references everything else is measured against: objectives, policies, acceptable risk, completion criteria and resource limits. A governance system makes execution accountable to those references. It does not decide, for an organization, what level of risk or cost is acceptable.

Frameworks and regulation

Standards specific to agents are still forming. NIST launched an AI Agent Standards Initiative in 2026, with agent security and identity among its research areas.

Getting started

A practical sequence, in which each step produces what the next one needs:

1. Inventory the agents in use, with an owner for each.
2. Classify each by autonomy and by what its actions can affect.
3. Bound authority with dedicated identities and least-privilege access.
4. Declare the task: have agents state the objective and scope of each run, so there is a reference to compare their actions with.
5. Designate high-consequence operations, and put human approval or other controls there, not everywhere.
6. Record attempted actions against the declared task as evidence, and keep it as long as accountability requires.
7. Define completion and verify outcomes against the systems acted on.
8. Review and adjust: measure departures, verified completion and cost per successful completion, and change policy based on what you find.

An example

The following is an illustrative scenario, not a record.

A finance operations team deploys an agent to reconcile supplier invoices against purchase orders each month.

Before. The finance systems lead approves the use, and the controller is named as accountable. The agent gets read access to the purchasing system and write access to reconciliation notes, nothing else. Any payment operation is designated high-consequence. Completion means every invoice is matched or flagged with a reason, within a resource limit per run.

During. The agent declares its objective and a scope covering the purchasing system. Midway it finds forty invoices without purchase order numbers and adapts, matching them by supplier and amount: a change of approach in service of the same objective. Later it attempts to email a supplier for a missing document. Email was never in scope, and the attempt is recorded as outside it.

After. The agent reports that reconciliation is complete. The team checks the purchasing system: three invoices are neither matched nor flagged. Execution finished and the agent claimed success, but the objective was not fully achieved. Cost data shows a third of the run’s tokens went to retrying one failing lookup.

Review. The team decides the supplier email was a reasonable idea and widens the scope to permit drafting, not sending. It adds a retry limit. It treats the three missed invoices as a completion failure, not a compliance one, and changes how the agent confirms its own results.

The authority and criteria came from people. The record of what was declared and attempted came from runtime governance. Verification came from the purchasing system, cost from usage data, and the judgments from review.

Where Sentience Governor fits

Sentience Governor implements the runtime governance and evidence part of this stack. It records each agent’s declared intent, evaluates each attempted action at the execution boundary against the declared scope, default policy and the operator’s governance profile, and keeps the findings as a local execution record. It observes and flags; every action proceeds. It does not enforce permissions, verify human authority, confirm that a task was completed, or see systems it is not instrumented on. Those parts of the stack belong to identity, authorization, verification and review, and Governor’s record is designed to be the evidence they draw on.

Often confused with

AI governance. The broader discipline for AI systems of every kind. Agent governance is the part that deals with authority, action and effect.

AI agent security. Security asks whether an agent, or someone controlling it, can misuse its access. Governance asks whether its actions are consistent with its task and the expectations that apply.

Agent observability. Observability makes a run inspectable. Governance decides what the run should be compared with, and who acts on the answer.

AI guardrails. One kind of control within the stack, checking defined limits at defined points.

Where the concept stops

AI agent governance is an organizational discipline supported by technical mechanisms. No product supplies it, and a complete set of controls does not supply it either without people who own the objectives, the policies and the consequences. Its evidence test is the six questions this article started with: when an agent has finished, can the organization say what it was authorized to do, what it was asked to do, what it attempted, what that achieved, what it cost, and who answers for it? Its practical test is what happens next: whether someone acts on those answers.