# Traces

> Reconstructing what an agent did: prompts, tool calls, arguments, results, and the data it touched.

Source: https://brocs.fyi/observe/traces/
Part of the surface: O · Observe. See what it is doing.
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---

## The Observe surface

- Metrics: golden signals plus tokens, cost, tool calls
- Traces: reconstruct what an agent did, with what, on what data
- Alerts: routed to the builder who owns the app
- What matters: signals that predict incidents, not dashboard decoration
- Model behavior: drift, silent provider updates, quality regressions


A trace is the ordered record of what happened inside one run. For a service, it is spans
across components. For an agent, it is the sequence of reasoning steps, tool calls,
arguments, results, and data touched, from the request to the final answer.

It is the artifact you need when someone asks what happened. It is also the artifact an
auditor will ask for, which makes it the most direct overlap between
[Observe](/observe/) and [Secure](/secure/).

## What a usable agent trace contains

- **The triggering input**, including who or what supplied it.
- **Each model call**: the model and version, the resolved prompt, the completion, the
  token counts, and the latency.
- **Each tool call**: the tool, the arguments, the result or error, and the identity the
  call was made under.
- **Every document retrieved**, with an identifier stable enough to find it again.
- **Decision points**: where the agent chose between options, and what it had available.
- **The terminal state**: completed, failed, stopped by a human, or abandoned on a timeout.

The test is whether a competent colleague, with no memory of the incident, can read the
trace and say what the system did and why. If they have to guess at a step, that step is
missing.

## Tool calls are the part that matters

Model reasoning is interesting. Tool calls are consequential. The tool call is where the
agent stops producing text and starts changing the world: writing a row, sending a
message, opening a ticket, deploying something.

So if you can only instrument one thing, instrument tool calls with full arguments,
results, and the identity used. That gives you the answer to "what did it actually do",
which is the question that gets asked in incidents and audits, and it is a shorter path
than capturing every token.

Two cautions. Arguments frequently contain credentials or personal data, so redact on the
way in rather than at query time. And record the identity the call was made under, not
just the tool name, because "the agent updated the record" and "the agent updated the
record as the finance service account" are very different sentences.

## Sampling, retention, and the honest trade

Full-fidelity traces on every request get expensive quickly, particularly when you are
storing prompts and completions.

A workable policy:

- **Sample the routine.** A small percentage of successful, low-risk sessions.
- **Keep everything abnormal.** Errors, human interventions, anything that touched
  production, anything above a cost threshold, anything a user flagged.
- **Keep tool calls at full fidelity even when you sample the reasoning.** They are small
  and they are the evidence.
- **Set retention by what it contains**, not by what it is. A trace holding personal data
  inherits that data's retention rules, and pretending it is telemetry does not change
  that.

## Correlation across the boundary

An agent trace that stops at the tool call boundary is half a story. The tool call hit a
service, that service hit a database, and the interesting failure is often on the far
side.

Propagate a trace identifier from the agent through every tool call into the downstream
systems, using whatever standard you already use. It is a small amount of plumbing and
it converts two separate investigations into one.

{{< failure id="cursor-support-bot" >}}

{{< aside title="How observability practitioners think about this" >}}
The distributed tracing discipline arrived at a settled answer: one trace identifier,
propagated everywhere, with spans that nest. Agents fit the model without modification;
a tool call is a span and a sub-agent is a child trace. The reason AI stacks so often end
up with a parallel, incompatible tracing system is that agent frameworks ship their own,
and nobody wires it to the existing one. That is an afternoon of work with a large
payoff.
{{< /aside >}}

{{< checklist >}}
- For an agent run last Tuesday, can you list every tool it called and with what arguments?
- Do traces record the identity each tool call was made under?
- What redacts credentials and personal data from tool arguments before storage?
- Does your agent trace share a trace identifier with your existing service tracing?
- What is the retention period on traces, and does it reflect what they contain?
- Which sessions do you keep at full fidelity, and who decided that list?
{{< /checklist >}}