Metrics

Metrics are the numbers that tell you whether an AI system is working, and for whom. The existing four golden signals still apply. They are no longer sufficient.

The four you already have

Latency, traffic, errors, and saturation. Keep all of them, with two amendments.

Latency needs percentiles and a time-to-first-token. A streaming response that starts in 400 milliseconds and finishes in nine seconds is a good experience. One that starts in eight seconds and finishes in nine is not, and total latency cannot tell them apart.

Saturation means someone else’s limit. The constrained resource is usually a provider rate limit or a token-per-minute quota you do not control. Track headroom against it, because the failure mode is a wall rather than a slope.

The ones that are new

Tokens in and out, per request. The base unit. Everything about cost and most things about latency derive from it.

Cost, attributed. Not the monthly invoice. Cost per request, aggregated by application, team, and user. This is the number that makes every conversation in Secure possible, and it can only be collected at the point of inference.

Tool calls per session. For agents, the strongest single indicator of health. A task that used to take four tool calls and now takes nineteen has degraded, and it will show up here before it shows up in a complaint.

Refusal and failure rate. How often the model declines, returns malformed output, or fails schema validation. A rising refusal rate usually means a prompt, a policy, or a provider changed underneath you.

Retry rate. Retries are where cost blows up quietly. An agent in a loop is a budget event.

Cache hit rate, if you use prompt caching. It moves cost by large multiples and it degrades silently when a prompt prefix changes.

Attribution is the design decision

The single most consequential choice in AI observability is whether every inference call carries an identity: which application, which team, which user, which session.

Add it at the start and every later question is answerable. Add it later and you are retrofitting identity into call sites across the estate, which in practice means it does not get added and cost stays a single line on an invoice.

This is the payoff for routing model access through a gateway rather than distributing provider keys. See keys and models.

Expose it to the people who own the thing

Central dashboards are necessary and insufficient. The person who built an internal tool should be able to see their own tool’s usage, cost, and error rate without asking the platform team, in the same way they can see its logs.

Two reasons. The obvious one is that they will notice a problem faster than a central team watching two hundred services. The less obvious one is that visible cost changes behaviour: a builder who can see that their nightly job costs 300 dollars a month will usually fix it without anyone opening a conversation about it.

The metric that is hardest and matters most

Task success rate. Did the thing the user wanted actually happen.

It is hard because it usually requires either an explicit signal from the user, an implicit one such as whether they retried or abandoned, or a judgement call from an evaluator. It is worth the effort because every other metric on this page can look healthy while the system is useless.