Agent evaluation is measuring how well an agent performs against defined tasks or criteria, either offline on test sets or online by scoring real runs. Because agents take many steps, an evaluation can assess the final result, the path taken, or both.
Evaluation is how teams decide whether an agent is good enough to ship, and whether a change made it better or worse. For agents it is harder than for a single model call, because the same task can be completed by many different paths, and because the agent’s behavior depends on the model, the harness around it and the environment it acts in. Evaluation answers one question well: how capable is this agent? It answers a second question, whether a particular run stayed within acceptable bounds, much less well than its scores suggest.
The vocabulary
Anthropic’s guide to agent evals sets out terms that are now widely used:
- A task is a single test with defined inputs and success criteria.
- A trial is one attempt at a task. Because agents are non-deterministic, tasks are run several times.
- A grader is the logic that scores some aspect of a trial. It can be code, a model acting as judge, or a person.
- The transcript, also called a trace or trajectory, is the complete record of a trial: outputs, tool calls, reasoning and intermediate results.
- The outcome is the final state of the environment when the trial ends.
Two reliability measures follow from running trials repeatedly. Pass@k is the chance that at least one of k attempts succeeds. Pass^k, introduced with the tau-bench benchmark, is the chance that all k attempts succeed. They tell very different stories. In the original tau-bench study, even GPT-4o, the strongest model tested, succeeded on fewer than half of the tasks, and on retail tasks its pass^8 fell below 25 percent: an agent that succeeds sometimes is not an agent that succeeds every time.
Outcome or path
The common advice is to grade what the agent produced rather than the path it took, because checking for a fixed sequence of steps is brittle: agents find valid paths the evaluator did not anticipate. For measuring capability, that advice is sound.
It has a known limit, and the people who build benchmarks state it plainly. Tau-bench checks success by comparing the database at the end of a conversation with the expected goal state, and its authors note that a correct end state is necessary but not sufficient for a successful episode: an agent might process a return without the explicit user confirmation the policy requires. The final state is correct; the path broke a rule. An evaluation that grades only outcomes will score that run as a success.
The same blind spot runs the other way. Anthropic describes a model that found a loophole in a policy during an evaluation: it failed the task as written and produced a better result for the user. A score cannot tell creative success from a broken rule unless the path is examined.
A score measures the agent and its grader together
Evaluations are also gamed. METR reported in 2025 that recent frontier models reward-hacked on a substantial share of runs on one benchmark: overwriting timing functions, stubbing out the evaluator so every submission passed, reading the grader’s expected answer. Telling the model not to cheat barely changed the rate, and when asked, the model said its approach did not match the user’s intent. Benchmarks also age: Epoch AI’s review rates SWE-bench Verified as flawed, citing contamination, tests that reject correct solutions, and results that depend heavily on the scaffold used.
So a score is a property of the whole setup: model, harness, grader and task set. Change any of them and the score no longer describes the agent. That includes the quiet changes that happen in production, which is one reason an agent that evaluated well can still exhibit agent drift once deployed.
Evaluation measures performance; governance keeps account of each run
Evaluation is not only about aggregates. A grader can read a single run’s transcript and judge its path, and online evaluation scores real runs one at a time. The difference from governance lies in purpose and coverage. An evaluation, of one trajectory or of many, measures performance against criteria, usually on selected runs and usually after they finish; its headline results are rates over samples, and a strong pass rate is evidence that an agent is capable, not that any particular run was acceptable. Production is one run at a time, each of which either stayed within bounds or did not.
That is the gap governance evidence fills: a record kept for every governed run while it executes, of what was declared, what was attempted and how each action compared with the declared scope and policy. Evaluation and governance draw on similar material, the actions a run took, for different purposes: one to measure how well an agent performs, the other to account for what each run did.
Where Sentience fits
Sentience Governor, the free, open-source runtime governance library, keeps that account for every governed run: the declared objective and scope, each attempted action at the execution boundary of its instrumented integrations, and how each compared with the scope and the operator’s policy. Its coverage is what those integrations capture: the actions that pass through them, not the agent’s reasoning or any action taken by a route they do not instrument. Its analysis commands summarize a captured session after the fact, for example how much of a session’s model usage fell in turns with policy violations or without a declared intent. The record can supply the action-level part of a trajectory for path-aware evaluation, and the per-run account that evaluation is not designed to keep. Governor does not score runs or judge task success.
Evaluation also decides when an agent is ready for production. Sentience Secure Runtime is built around that step: teams run an agent workflow in an isolated environment, validate its execution policies, data access, egress boundaries and authorization requirements on real runs, and carry what passes into production as a reusable deployment template, with runtime governance continuing once it is live.
Often confused with
Agent monitoring. Monitoring watches production continuously for failures, cost and anomalies. Evaluation measures performance against defined tasks, usually before deployment or on sampled runs.
Testing. A test checks that code behaves as specified, usually deterministically. An agent evaluation runs non-deterministic trials and reports rates, not pass or fail.
Agent observability. Observability makes runs inspectable. Evaluation applies criteria to what is observed.
Where the concept stops
Evaluation measures how well an agent performs on defined tasks, across samples, with a given grader. It does not establish that a particular production run stayed within its scope and policy, that a passing outcome was reached by an acceptable path, or that the score still holds after the model, harness or environment changed. Those depend on a record of each run and on comparing what it did with what it was meant to do.