# What matters

> Which signals predict incidents and which are dashboard decoration, and how to expose observability to people who are not on the platform team.

Source: https://brocs.fyi/observe/what-matters/
Part of the surface: O · Observe. See what it is doing.
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---

## The Observe surface

- Metrics: golden signals plus tokens, cost, tool calls
- Traces: reconstruct what an agent did, with what, on what data
- Alerts: routed to the builder who owns the app
- What matters: signals that predict incidents, not dashboard decoration
- Model behavior: drift, silent provider updates, quality regressions


Most AI observability effort goes into building a dashboard that looks impressive in a
review and is never opened during an incident. This page is about the difference.

## Decoration and signal

A metric is decoration if you cannot name a decision it would change.

Token counts by model, on a large chart, are decoration in most organizations. Nobody
looks at that chart and does something differently. Cost per session for the three
applications that account for 80 percent of spend is a signal, because there is an
obvious action attached.

Run the test on your existing dashboard, honestly. For each panel: what decision does
this change, and who makes it? Panels that survive are your dashboard. The rest can go in
a query you run when you need it.

## The four questions worth instrumenting for

Most useful AI observability answers one of these:

1. **Is it working for users right now?** Task success rate, error rate,
   time-to-first-token. The ones that belong in an objective.
2. **Is it getting worse?** Eval scores over time, refusal rate, tool calls per task,
   user-reported quality. Trend rather than threshold.
3. **What did it do?** [Traces](/observe/traces/). Asked after the fact, usually by
   someone with a deadline.
4. **What is it costing, and who is causing it?** Attributed cost. Asked monthly by
   finance and continuously by anyone with a budget.

If a signal does not serve one of those four, it is context. Keep it, do not chart it.

## Leading and lagging

Error rate is lagging: by the time it moves, users are affected. The useful work is
finding indicators that move first.

In practice the reliable leading indicators are:

- **Tool calls per completed task**, rising. The agent is working harder for the same
  result, which usually means retrieval quality or instruction quality has degraded.
- **Retry rate**, rising. Something downstream is flaky, or output validation is failing
  more often.
- **Prompt length**, growing. Context accumulation, usually from a change nobody flagged,
  and it moves cost and latency together.
- **Eval score on the pinned suite**, falling. The most direct one, and the only one that
  needs deliberate investment.

## Expose it beyond the platform team

The people who most need AI observability are frequently the least likely to have access
to it: the analyst who built the tool, the product manager who owns the workflow, the
compliance officer who has to sign something.

Three surfaces, and they are not the same surface:

**The builder view.** My application: cost, errors, recent traces, eval results. Scoped to
what I built, no query language required.

**The owner view.** This workflow: task success rate, volume, cost trend, user-reported
problems. Aimed at someone who cares whether it works, not how.

**The evidence view.** Given a date range and a system, produce the record: what ran, what
it touched, which approvals applied. Aimed at audit, and covered in
[Secure](/secure/compliance/).

Building only the first is the common failure. It leaves the people who decide whether to
keep funding the thing with no way to see whether it works.

{{< aside title="How control theory thinks about this" >}}
A control loop needs a measurement, a setpoint, and an actuator. Most AI monitoring
supplies a measurement and nothing else: no defined target, and no mechanism to adjust
anything without a deploy. That is not a loop, it is a thermometer. When you add a metric,
name the setpoint and the actuator at the same time. If either is missing, you are
building decoration, however good the chart looks.
{{< /aside >}}

{{< checklist >}}
- For each panel on your main AI dashboard: what decision does it change, and whose?
- Which of your metrics is leading rather than lagging?
- Can the person who built an application see its cost and traces themselves?
- Is there a view aimed at someone who is not technical and owns the outcome?
- When was your dashboard last opened during an actual incident?
- What is the setpoint for your most-watched metric?
{{< /checklist >}}