Agent observability is the ability to inspect and understand an agent's execution through recorded data such as traces, tool calls, inputs, outputs, timing and errors. It provides visibility into what occurred and the surrounding execution context, while judging an action's appropriateness requires criteria against which to evaluate it.
Observability is the discipline of making a system's internal behavior understandable from the signals it emits. Applied to an AI agent, it has more to explain than whether a service is up. Because an agent selects its operations dynamically, the question is how a particular execution unfolded: across model calls, tools, external systems and, often, a long sequence of steps that nobody wrote down in advance.
What it extends
Conventional application observability rests on three signals: traces that follow a request through a system, metrics that aggregate behavior over time, and logs that record discrete events. All three carry over to agents. What changes is the thing being traced.
| Conventional service | Agent |
|---|---|
| A request follows code paths a developer wrote | A run follows a path the model chose |
| Spans are function calls, queries, outbound requests | Spans are also model invocations, tool executions and agent handoffs |
| The main questions are latency, errors and saturation | Those remain, joined by what the model was given, what it chose, and what that cost |
| Two identical requests usually produce similar traces | The same task can produce different traces on different runs |
OpenTelemetry's semantic conventions for generative AI describe this shape in a vendor-neutral way. They define operations such as invoking an agent, calling a model and executing a tool, and attributes for the model, the agent, the conversation, the tool and token usage. They are still marked as in development, and the major agent frameworks and platforms ship their own tracing alongside or on top of them.
What gets recorded
- Traces and spans. A run is a tree: the agent invocation at the root, model calls and tool executions beneath it, nested agents beneath those. Each span has a start, a duration, a status and attributes.
- Model usage. Input and output token counts per call, the model requested and the model that answered, and cache behavior where the provider reports it.
- Tool-call records. Which tool, how long it took, whether it failed, and, where the instrumentation is configured to keep them, its arguments and result.
- Errors and retries. Exceptions, timeouts, rate limits, and the repeated attempts that follow them.
- Latency. Per span and end to end, including time spent waiting on a person.
- Metrics. Aggregates over many runs: token usage, operation duration, error rates, tool usage.
- Logs and events. Discrete occurrences that do not fit a span, and, where enabled, the content of prompts and responses.
- The execution sequence. The order and nesting of all of the above, which is what turns a pile of spans into an account of a run.
Why individual model calls are not enough
Instrumenting each model call tells you that a call happened, how long it took and how many tokens it used. It does not tell you that calls seven through eleven were the agent retrying the same failing tool, or that the expensive call at the end re-read forty thousand tokens of history that an earlier step had made irrelevant.
The unit of understanding for an agent is the run. That requires correlation: a shared trace across model calls and tool executions, a conversation or session identifier that survives across turns, and parent and child relationships that show which decision led to which operation. Without them a multi-step execution looks like a list of unrelated requests.
Live and retrospective
Observability is not something that happens afterward. Spans and metrics are emitted as the run proceeds, and the same data serves different uses at different times.
- While the agent runs: dashboards, alerts on error rates or spend, live inspection of a run that appears stuck, and signals that other systems can act on.
- After it finishes: debugging a failure, comparing runs, tracing a regression to a prompt or a model change, building evaluation datasets from real traffic, and capacity and cost planning.
Agent monitoring is the live, threshold-driven end of this, and agent tracing is the mechanism that produces the run-level picture. Observability is the broader capability that both serve.
The economics of execution
Agents make cost a first-order observability concern, because the cost of a task is not the cost of a call.
The measurable inputs are tokens by class, tool invocations, retries, elapsed time and, where price data is joined in, money. What makes them meaningful is the denominator. Cost per model call rewards small calls. Cost per run is better. Cost per successful completion is better again, because it charges the retries, the failed runs, the corrections and the human interventions to the outcomes that were achieved. That figure needs something most traces do not contain by themselves: a record of whether the task succeeded. It usually has to be joined in from evaluation, from the systems acted on, or from a person. See AI agent cost.
Privacy and the instrumentation tradeoff
The most useful observability data is also the most sensitive. Prompts contain user input, retrieved documents and source code. Tool arguments contain identifiers and sometimes credentials. Tool results contain whatever the system returned.
Instrumentation therefore involves a real choice.
| Approach | What you gain | What you take on |
|---|---|---|
| Record prompts, completions, arguments and results | Detailed reconstruction and debugging of the recorded execution, including what the model was given | Sensitive data in a telemetry pipeline, with its own access control, retention and residency obligations |
| Record structured metadata only | Timing, usage, tool identity, status and sequence, at much lower risk | Less ability to explain why a run went the way it did |
OpenTelemetry's conventions reflect this. The attributes that carry message content and tool arguments and results are opt-in, and the specification warns that they are likely to contain sensitive information. Most platforms offer redaction, sampling or content-free modes for the same reason. Neither choice is wrong. It depends on what the data is and who may see it.
What observability does not see
- Uninstrumented operations. A tool that is not wrapped, a subprocess the agent starts, an action a provider performs on its own infrastructure. Coverage is a property of instrumentation, not of the run.
- Unavailable results. A span can record that a tool was called and returned without retaining what it returned.
- Effects. A span that ends successfully shows that a call returned. It does not show what changed in the target system. An observed attempt is not a verified effect, and confirming one takes evidence from the system that was acted on.
Observability and runtime governance
These two are often confused, and the distinction is worth drawing carefully because it is not a contest.
Observability establishes what can be seen and understood about an execution. Runtime governance establishes one explicit relationship: between a declared purpose, an attempted action, the applicable policy and the finding of an evaluation. AI agent governance is the wider discipline that both serve.
Nothing stops an observability platform from carrying that relationship. An objective can be a span attribute. A policy decision can be an event. Governance findings can be exported as telemetry, and the two can live in one architecture. What observing an execution does not do, by itself, is establish task-relative governance. A trace holds what its instrumentation and retention settings provide, which can be a great deal. Visibility into execution does not, however, automatically establish an evaluated relationship to a declared purpose. That reference has to be stated, the comparison has to be made, and the result has to be kept. When a platform is instrumented to do those three things, it is doing governance, inside an observability system.
The same scenario, two kinds of record
Consider the coding agent scenario used elsewhere in this glossary: an agent asked to fix a failing test, which edits a file, runs the tests and requests the deletion of two directories.
The two records below were not collected from one live execution, and should not be read that way. The first is an illustrative sketch of what an observability trace of such a run could look like, using OpenTelemetry's generative AI operation names, with invented figures. The second is genuine Sentience Governor output from a separately generated demonstration of the same scenario. They are set side by side to show the different questions each kind of record answers.
The illustrative trace:
invoke_agent coding-agent 41.2 s
├─ chat example-model in 12,480 out 310 3.1 s
├─ execute_tool Bash pytest ... 6.8 s status: error (exit 1)
├─ chat example-model in 13,020 out 540 4.0 s
├─ execute_tool Edit src/dates.py 0.1 s
├─ chat example-model in 13,700 out 120 2.2 s
├─ execute_tool Bash rm -rf .cache build/ 0.3 s
├─ chat example-model in 13,950 out 95 2.0 s
└─ execute_tool Bash pytest ... 7.1 s status: okThe genuine governance event, abridged from Sentience Governor 0.3.2 output generated from representative Claude Code hook payloads for the command rm -rf .cache build/. No command was executed in producing it, and no directory was deleted:
{
"event_type": "SCOPE_ASSERTED",
"event_sequence_number": 11,
"profile_fingerprint": "a99fbe0cdb5a",
"advisory_flags": ["HIGH_CONSEQUENCE_DETECTED"],
"pass_through": true,
"payload": {
"tool_id": "Bash",
"operation_type": "EXECUTE",
"target_system": "shell/filesystem",
"operation_classification": { "complete": true, "destructive": true }
}
}Each answers questions the other does not.
| Question | Trace | Governance record |
|---|---|---|
| How long did the run take, and where did the time go? | Yes | No |
| How many tokens did each model call use? | Yes | Per turn, where the integration reports usage |
| Which command was requested, with which arguments? | Yes, if content capture is on | No. Executable name and classification only |
| What did the tests print? | Yes, if results are retained | No |
| What was the agent supposed to be doing? | Only if instrumented to carry it | Yes, as a declared objective with its source |
| Was the deletion within the declared scope, and what did policy say about it? | Only if instrumented to carry it | Yes |
| Which policy was in force? | Only if instrumented to carry it | Yes, by fingerprint |
| Were the directories actually deleted? | No | No |
The last row matters as much as the others. Neither kind of record confirms the effect.
Where Sentience Governor fits
Sentience Governor is not an observability platform, and it does not replace one. It records execution evidence of a specific kind.
In its released 0.3.2 form it records a declared intent and its source, each attempted action at the execution boundary where it is instrumented, the findings of a deterministic evaluation against scope and policy, and the identity of the policy applied. Where an integration reports usage, it attributes model tokens to turns in four classes, cached read, cached write, prompt and completion, and can report how much usage fell on turns that carried findings. This is governance evidence, organized around a declared intent.
Measured against the list above, it is deliberately narrow. It does not capture prompts or completions. It does not retain tool arguments or tool results, keeping structured metadata and, for shell commands, executable names and a classification. It records event timestamps and not span durations or latency metrics. It does not observe system-level effects. It reports tokens and does not calculate monetary cost, and it has no knowledge of whether a task succeeded, so it does not produce a cost per successful completion. It does not emit OpenTelemetry data. It writes to a local record, and an operator can configure a sink that forwards events elsewhere, which is how its records can sit beside a trace in a platform built for that.
Where the concept stops
Observability makes execution visible and explicable. It does not decide what an agent should have been doing, and it does not confirm what changed in the world. Its reach is set by instrumentation, and its depth by how much content an organization is willing to keep. Within those limits it is the basis for nearly everything else: debugging, cost control, evaluation, and the review that governance depends on.