Observe
Metrics, traces, alerts, and quality measurements for AI systems, available to the people who own them.
Observe covers the metrics, traces, alerts, and quality measurements used to operate AI systems. These signals need to reach the people who own each application.
A conventional monitoring stack covers infrastructure and request failures. AI systems also need a direct measure of output quality.
Quality signals
An AI system can return the same status code and latency while its answers degrade. Infrastructure dashboards will remain stable during that failure.
Measure quality through evaluation canaries and user reports. Keep traces detailed enough to reconstruct agent activity during an incident or audit.
Operating tests
- Produce last month's cost for one application from internal records.
- Reconstruct every tool call and argument from a specific agent run last week.
- Test whether internal measurements detect a 20 percent quality drop before a customer reports it.
Telemetry controls
Detailed traces may contain prompts and retrieved content, including tool arguments. Apply Secure redaction, retention, and access controls to telemetry at collection time.
Connect actionable measurements to the relevant Control policy or operating procedure.
In this capability
- Metrics: Golden signals plus the AI signals: tokens, cost, tool calls, and refusal rate, attributed per app, per team, and per user.
- Traces: Reconstructing what an agent did: prompts, tool calls, arguments, results, and the data it touched.
- Alerts: Routing application alerts to the builder or platform team that owns the response.
- Useful signals: Signals tied to operating decisions, including access for application owners and audit teams.
- Model behaviour: Detecting model drift and provider-driven quality regressions that infrastructure monitoring does not show.