Traces
Reconstructing what an agent did: prompts, tool calls, arguments, results, and the data it touched.
A trace is the ordered record of one run. A service trace contains spans across components. An agent trace also records model activity, tool calls, results, and data touched from the request to the final answer.
The trace supports incident reconstruction and audit evidence, so its collection sits between Observe and Secure.
Agent trace fields
- Record the triggering input and its source.
- For each model call, record the model and version, resolved prompt, completion, token counts, and the latency.
- For each tool call, record the tool, arguments, result or error, and the identity the call was made under.
- Record every retrieved document under a stable identifier.
- Record decision points and the options available to the agent.
- Record the terminal state, including completion, failure, human stop, or timeout.
A colleague unfamiliar with the incident should be able to reconstruct the system's actions from the trace. Missing steps should be visible as gaps in the record.
Tool-call records
Tool calls create external effects such as database writes, messages, tickets, and deployments. Give these records priority when trace storage is limited.
Capture tool arguments, results, and execution identity. These fields answer the central incident and audit question about actions taken by the system.
Tool arguments may contain credentials or personal data, so redact them before storage. Keep the execution identity with the tool name because permissions depend on that identity.
Sampling and retention
Full-fidelity traces on every request get expensive quickly, particularly when you are storing prompts and completions.
- Sample a small percentage of successful low-risk sessions.
- Keep errors, human interventions, anything that touched production, anything above a cost threshold, anything a user flagged.
- Keep tool calls at full fidelity when possible, even if model activity is sampled.
- Set retention from the data contained in the trace. A trace with personal data inherits the applicable retention rules.
End-to-end correlation
An agent trace should continue through downstream services and databases. Otherwise an investigation needs to join separate records by timestamps and payload details.
Propagate a trace identifier from the agent through every tool call into the downstream systems using the organization's existing tracing standard.
A support bot invented a policy and customers cancelled over it
Users switching machines started getting logged out. Cursor's AI support agent, which signed its emails as Sam, told them this was a deliberate one-device-per-subscription security policy. No such policy existed. Developers cancelled.
The skipped letter. Observe. Nothing sampled what the agent was actually telling customers, so a fabricated policy circulated at scale until customers escalated it on social media. Your users should not be the monitoring layer.
eWeek: Cursor AI chatbot hallucination, fake policy / CX Today: a contact centre chatbot invents company policies
Questions to ask your vendor, or your own team
- For an agent run last Tuesday, can you list every tool it called and with what arguments?
- Do traces record the identity each tool call was made under?
- What redacts credentials and personal data from tool arguments before storage?
- Does your agent trace share a trace identifier with your existing service tracing?
- What is the retention period on traces, and does it reflect what they contain?
- Which sessions do you keep at full fidelity, and who decided that list?