# Metrics

> Golden signals plus the AI signals: tokens, cost, tool calls, and refusal rate, attributed per app, per team, and per user.

Source: https://brocs.fyi/observe/metrics/
Part of the surface: O · Observe. See what it is doing.
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---

## The Observe surface

- Metrics: golden signals plus tokens, cost, tool calls
- Traces: reconstruct what an agent did, with what, on what data
- Alerts: routed to the builder who owns the app
- What matters: signals that predict incidents, not dashboard decoration
- Model behavior: drift, silent provider updates, quality regressions


Metrics are the numbers that tell you whether an AI system is working, and for whom. The
existing four golden signals still apply. They are no longer sufficient.

## The four you already have

Latency, traffic, errors, and saturation. Keep all of them, with two amendments.

**Latency needs percentiles and a time-to-first-token.** A streaming response that starts
in 400 milliseconds and finishes in nine seconds is a good experience. One that starts in
eight seconds and finishes in nine is not, and total latency cannot tell them apart.

**Saturation means someone else's limit.** The constrained resource is usually a provider
rate limit or a token-per-minute quota you do not control. Track headroom against it,
because the failure mode is a wall rather than a slope.

## The ones that are new

**Tokens in and out, per request.** The base unit. Everything about cost and most things
about latency derive from it.

**Cost, attributed.** Not the monthly invoice. Cost per request, aggregated by
application, team, and user. This is the number that makes every conversation in
[Secure](/secure/cost/) possible, and it can only be collected at the point of inference.

**Tool calls per session.** For agents, the strongest single indicator of health. A task
that used to take four tool calls and now takes nineteen has degraded, and it will show
up here before it shows up in a complaint.

**Refusal and failure rate.** How often the model declines, returns malformed output, or
fails schema validation. A rising refusal rate usually means a prompt, a policy, or a
provider changed underneath you.

**Retry rate.** Retries are where cost blows up quietly. An agent in a loop is a budget
event.

**Cache hit rate**, if you use prompt caching. It moves cost by large multiples and it
degrades silently when a prompt prefix changes.

## Attribution is the design decision

The single most consequential choice in AI observability is whether every inference call
carries an identity: which application, which team, which user, which session.

Add it at the start and every later question is answerable. Add it later and you are
retrofitting identity into call sites across the estate, which in practice means it does
not get added and cost stays a single line on an invoice.

This is the payoff for routing model access through a gateway rather than distributing
provider keys. See [keys and models](/build/keys-and-models/).

## Expose it to the people who own the thing

Central dashboards are necessary and insufficient. The person who built an internal tool
should be able to see their own tool's usage, cost, and error rate without asking the
platform team, in the same way they can see its logs.

Two reasons. The obvious one is that they will notice a problem faster than a central
team watching two hundred services. The less obvious one is that visible cost changes
behaviour: a builder who can see that their nightly job costs 300 dollars a month will
usually fix it without anyone opening a conversation about it.

{{< aside title="How SRE thinks about this" >}}
The SRE discipline separates the signals users feel from the signals operators care
about, and insists the first set drives the alerts. Applied here: tokens and cost are
operator metrics, and they should not page anyone. Time-to-first-token, error rate, and
task completion rate are user metrics, and those are the ones that belong in an
objective. The trap is building a beautiful token dashboard and never measuring whether
the thing actually worked.
{{< /aside >}}

## The metric that is hardest and matters most

Task success rate. Did the thing the user wanted actually happen.

It is hard because it usually requires either an explicit signal from the user, an
implicit one such as whether they retried or abandoned, or a judgement call from an
evaluator. It is worth the effort because every other metric on this page can look
healthy while the system is useless.

{{< checklist >}}
- Can you produce cost for one application, last month, without asking a vendor?
- Does every inference call carry an application, team, and user identity?
- What is your time-to-first-token at the 95th percentile?
- Do you track tool calls per agent session, and do you know what normal looks like?
- Can a builder see their own application's metrics without filing a request?
- What is your task success rate, and if you cannot answer, what would it take?
{{< /checklist >}}