# Alerts

> Routing alerts to the builder who owns the application, not only to a central operations queue.

Source: https://brocs.fyi/observe/alerts/
Part of the surface: O · Observe. See what it is doing.
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---

## The Observe surface

- Metrics: golden signals plus tokens, cost, tool calls
- Traces: reconstruct what an agent did, with what, on what data
- Alerts: routed to the builder who owns the app
- What matters: signals that predict incidents, not dashboard decoration
- Model behavior: drift, silent provider updates, quality regressions


An alert is a claim that a human needs to do something now. Most AI monitoring produces
the opposite: dashboards nobody watches and thresholds that fire into a channel with
four hundred members.

## Route to the owner, not the centre

The organizing principle is that the alert goes to whoever can fix the thing. For an
internal tool built by an analyst, that is the analyst. For a shared inference gateway,
that is the platform team.

This sounds obvious and is rarely implemented, because AI applications frequently arrive
without an owner field. Somebody built it, it works, it is in production, and the
monitoring defaults route to the platform team because that is where monitoring lives.

The platform team then receives alerts about applications they did not build, cannot
change, and do not understand. They triage by forwarding. Everyone learns the alerts are
not actionable, and the next real one is missed.

Fix it at provisioning time: an application without a named owner does not get created.
See [idea to production](/build/idea-to-production/).

## What is worth waking someone for

Three categories, and it is worth being ruthless about the boundary.

**Page a human.** The system is failing users right now and a person must intervene:
sustained error rate, provider outage with no fallback, an agent stuck in a loop burning
budget, a security control that failed open.

**Ticket for the morning.** Something is wrong but tolerable: elevated refusal rate, a
cost anomaly within the budget, a single failed evaluation, rising latency with no
breaches.

**Dashboard only.** Everything else. Token counts, cache hit rates, and volume are context
during an investigation, not events.

Most organizations put the second category in the first, discover the pager is noisy, and
then stop trusting it. Moving things down is harder than never putting them there.

## AI-specific alerts worth having

- **Cost rate of change.** Not a monthly total. A sudden step in spend per hour catches a
  retry loop in minutes rather than at invoice time.
- **Agent session duration outliers.** A session running four times longer than the
  median is usually stuck, and it is usually still spending.
- **Refusal or schema-failure spike.** Almost always means something upstream changed:
  a prompt, a policy, a model version.
- **Eval regression on the pinned suite.** The closest thing to a quality alert. See
  [model behaviour](/observe/model-behavior/).
- **Guardrail trips.** If a filter or policy check starts firing at ten times its
  baseline, someone is probing or something broke.
- **Silence.** A scheduled AI job that produced no output is a failure that looks like
  nothing. Alert on absence.

## The alert that should exist and usually does not

Quality degradation reported by a human. Give users a one-click way to flag a bad
response, route those flags somewhere a person reads, and treat a cluster of them as an
incident signal.

It is unfashionable because it is not automated. It is also, in most organizations, the
fastest detector you have for the failure mode traditional monitoring cannot see: a
system returning 200s with worse answers than last week.

{{< aside title="How on-call practice thinks about this" >}}
The settled wisdom is that every page must be actionable, novel, and urgent, and that
alerts which do not meet the bar should be deleted rather than muted. The AI amendment is
that a whole class of real failures is not detectable by threshold at all, which means
the alert has to come from an eval, a canary, or a user. Building those signals is the
work; tuning thresholds is not.
{{< /aside >}}

{{< checklist >}}
- For each AI application in production, who gets paged? Name them.
- How many AI alerts fired last month, and how many resulted in an action?
- Do you alert on cost rate of change, or only on totals?
- What happens when a scheduled AI job produces nothing at all?
- Can a user report a bad response in one click, and where does that go?
- Is there an alert that would have caught your last AI incident? If not, is there one now?
{{< /checklist >}}