# Observe

> Metrics, traces, and alerts for AI systems, exposed to the people who own them. Including the failures that return a 200 and look like nothing.

Source: https://brocs.fyi/observe/
Part of the surface: O · Observe. See what it is doing.
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---

## The Observe surface

- Metrics: golden signals plus tokens, cost, tool calls
- Traces: reconstruct what an agent did, with what, on what data
- Alerts: routed to the builder who owns the app
- What matters: signals that predict incidents, not dashboard decoration
- Model behavior: drift, silent provider updates, quality regressions


**Observe** is the part of the surface concerned with seeing what your AI systems are
actually doing: the numbers that say whether it works, the traces that say what it did,
the alerts that reach the right person, and the measurements that catch quality moving
underneath you.

It is the letter organizations most often think they have covered because they have a
monitoring stack. The monitoring stack is necessary and it will not catch the failure
that matters most here.

## The claim

Traditional monitoring detects systems that stop working. The characteristic AI failure
is a system that keeps working and gets worse: the same latency, the same status code, a
worse answer. Nothing on a conventional dashboard moves.

That has two consequences. The first is that quality needs its own measurement, which
means [evals](/observe/model-behavior/), canaries, and a way for users to flag a bad
answer. The second is that [traces](/observe/traces/) are not a debugging luxury. They
are the only record of what an agent did, and they are what an auditor will ask for.

## The tests that matter

1. **Cost, attributed.** Can you say what one application cost last month, without asking
   a vendor?
2. **Reconstruction.** For a specific agent run last week, can you list every tool call
   and every argument?
3. **Who finds out first.** If output quality dropped 20 percent tomorrow, would it be
   your measurement or your customer?

## The tension

Observe pulls against [Secure](/secure/) more than people expect. The most useful trace is
the most complete one, and the most complete one is full of prompts, tool arguments, and
retrieved documents, which is to say full of exactly the data you are trying to protect.
Redaction, retention, and access control on the telemetry itself are part of the job
rather than an afterthought.

Observe hands off to [Control](/control/): measurement without an actuator is a
thermometer, not a loop.




## Pages under Observe

- [Metrics](https://brocs.fyi/observe/metrics/): Golden signals plus the AI signals: tokens, cost, tool calls, and refusal rate, attributed per app, per team, and per user.
- [Traces](https://brocs.fyi/observe/traces/): Reconstructing what an agent did: prompts, tool calls, arguments, results, and the data it touched.
- [Alerts](https://brocs.fyi/observe/alerts/): Routing alerts to the builder who owns the application, not only to a central operations queue.
- [What matters](https://brocs.fyi/observe/what-matters/): Which signals predict incidents and which are dashboard decoration, and how to expose observability to people who are not on the platform team.
- [Model behaviour](https://brocs.fyi/observe/model-behavior/): Drift, silent provider updates, and quality regressions: the failures that return a 200 and look like nothing on a dashboard.