What matters

Most AI observability effort goes into building a dashboard that looks impressive in a review and is never opened during an incident. This page is about the difference.

Decoration and signal

A metric is decoration if you cannot name a decision it would change.

Token counts by model, on a large chart, are decoration in most organizations. Nobody looks at that chart and does something differently. Cost per session for the three applications that account for 80 percent of spend is a signal, because there is an obvious action attached.

Run the test on your existing dashboard, honestly. For each panel: what decision does this change, and who makes it? Panels that survive are your dashboard. The rest can go in a query you run when you need it.

The four questions worth instrumenting for

Most useful AI observability answers one of these:

1. Is it working for users right now? Task success rate, error rate, time-to-first-token. The ones that belong in an objective.
2. Is it getting worse? Eval scores over time, refusal rate, tool calls per task, user-reported quality. Trend rather than threshold.
3. What did it do? Traces. Asked after the fact, usually by someone with a deadline.
4. What is it costing, and who is causing it? Attributed cost. Asked monthly by finance and continuously by anyone with a budget.

If a signal does not serve one of those four, it is context. Keep it, do not chart it.

Leading and lagging

Error rate is lagging: by the time it moves, users are affected. The useful work is finding indicators that move first.

In practice the reliable leading indicators are:

• Tool calls per completed task, rising. The agent is working harder for the same result, which usually means retrieval quality or instruction quality has degraded.
• Retry rate, rising. Something downstream is flaky, or output validation is failing more often.
• Prompt length, growing. Context accumulation, usually from a change nobody flagged, and it moves cost and latency together.
• Eval score on the pinned suite, falling. The most direct one, and the only one that needs deliberate investment.

Expose it beyond the platform team

The people who most need AI observability are frequently the least likely to have access to it: the analyst who built the tool, the product manager who owns the workflow, the compliance officer who has to sign something.

Three surfaces, and they are not the same surface:

The builder view. My application: cost, errors, recent traces, eval results. Scoped to what I built, no query language required.

The owner view. This workflow: task success rate, volume, cost trend, user-reported problems. Aimed at someone who cares whether it works, not how.

The evidence view. Given a date range and a system, produce the record: what ran, what it touched, which approvals applied. Aimed at audit, and covered in Secure.

Building only the first is the common failure. It leaves the people who decide whether to keep funding the thing with no way to see whether it works.