Metrics

Metrics show whether an AI system is working and which users it serves. Keep the existing golden signals and add measures for model usage, agent activity, and task outcomes.

Infrastructure signals

Continue to track latency, traffic, errors, and saturation.

Latency needs percentiles and a time-to-first-token. Total latency cannot distinguish a response that begins streaming in 400 milliseconds from one that begins after eight seconds when both finish at nine seconds.

Saturation may be a provider rate limit or a token-per-minute quota you do not control. Track headroom because requests fail once the quota is exhausted.

AI-specific signals

Track input and output tokens per request. They drive model cost and influence latency.

Attribute request cost to the application, team, and user at the point of inference. Attributed cost supports the controls in Secure.

Track tool calls per agent session. An increase for the same task can indicate degraded retrieval or instructions before users report a problem.

Measure refusal, malformed output, and schema-validation failures. A rate change may follow an update to a prompt, policy, or provider model.

Measure retry rate because loops increase both load and cost.

Measure prompt-cache hit rate where caching is enabled. A changed prompt prefix can lower the rate and increase cost.

Call attribution

Every inference call should carry the application, team, user, and session identity.

Add these fields when the gateway or client integration is built. Retrofitting identity across existing call sites requires more coordination and leaves historical costs unattributed.

Routing model access through a gateway makes this attribution available.

Metrics for application owners

Give application owners direct access to their tool's usage, cost, error rate, and logs. Keep the central dashboard for platform-wide operations.

Owners can investigate a problem sooner when they have the data. Visible cost also lets them tune a job before a platform team starts a separate review.

Task success

Task success rate measures whether the user obtained the intended result.

It may require an explicit user signal, an implicit signal such as retry or abandonment, or an evaluator judgement. Compare it with infrastructure metrics because a technically healthy service can still fail the user's task.