Alerts

An alert is a claim that a human needs to do something now. Most AI monitoring produces the opposite: dashboards nobody watches and thresholds that fire into a channel with four hundred members.

Route to the owner, not the centre

The organizing principle is that the alert goes to whoever can fix the thing. For an internal tool built by an analyst, that is the analyst. For a shared inference gateway, that is the platform team.

This sounds obvious and is rarely implemented, because AI applications frequently arrive without an owner field. Somebody built it, it works, it is in production, and the monitoring defaults route to the platform team because that is where monitoring lives.

The platform team then receives alerts about applications they did not build, cannot change, and do not understand. They triage by forwarding. Everyone learns the alerts are not actionable, and the next real one is missed.

Fix it at provisioning time: an application without a named owner does not get created. See idea to production.

What is worth waking someone for

Three categories, and it is worth being ruthless about the boundary.

Page a human. The system is failing users right now and a person must intervene: sustained error rate, provider outage with no fallback, an agent stuck in a loop burning budget, a security control that failed open.

Ticket for the morning. Something is wrong but tolerable: elevated refusal rate, a cost anomaly within the budget, a single failed evaluation, rising latency with no breaches.

Dashboard only. Everything else. Token counts, cache hit rates, and volume are context during an investigation, not events.

Most organizations put the second category in the first, discover the pager is noisy, and then stop trusting it. Moving things down is harder than never putting them there.

AI-specific alerts worth having

• Cost rate of change. Not a monthly total. A sudden step in spend per hour catches a retry loop in minutes rather than at invoice time.
• Agent session duration outliers. A session running four times longer than the median is usually stuck, and it is usually still spending.
• Refusal or schema-failure spike. Almost always means something upstream changed: a prompt, a policy, a model version.
• Eval regression on the pinned suite. The closest thing to a quality alert. See model behaviour.
• Guardrail trips. If a filter or policy check starts firing at ten times its baseline, someone is probing or something broke.
• Silence. A scheduled AI job that produced no output is a failure that looks like nothing. Alert on absence.

The alert that should exist and usually does not

Quality degradation reported by a human. Give users a one-click way to flag a bad response, route those flags somewhere a person reads, and treat a cluster of them as an incident signal.

It is unfashionable because it is not automated. It is also, in most organizations, the fastest detector you have for the failure mode traditional monitoring cannot see: a system returning 200s with worse answers than last week.