Alerts

An alert should identify a condition that needs prompt human action. Dashboards and large notification channels are useful for context, but they do not replace an owned response.

Owner-based routing

Route each alert to the person or team that can change the affected system. An analyst may own an internal tool, while the platform team owns a shared inference gateway.

Capture the owner when the application is provisioned. Otherwise monitoring defaults may send every alert to the platform team.

Forwarding alerts adds delay and trains recipients to ignore the channel. The owning team needs the notification and the permissions required to respond.

Require an owner during the idea-to-production workflow before creating the application record.

Alert severity

• Page when the system is failing users and a person must intervene, such as a sustained error rate, a provider outage with no fallback, an agent loop burning budget, or a security control that failed open.
• Create a ticket for tolerable problems that need follow-up, such as elevated refusal rate, a cost anomaly within budget, a failed evaluation, or rising latency below the objective.
• Keep token counts, cache hit rates, and request volume on dashboards for investigation context.

When routine issues page the on-call team, noise grows and trust in the pager falls. Set the initial boundary conservatively and review it after incidents.

AI-specific alerts

• Alert on a sudden change in spend per hour to catch a retry loop before invoice time.
• Detect agent session duration outliers that may indicate a stuck process.
• Track spikes in refusal and schema-validation failures after prompt, policy, or model changes.
• Alert when the pinned model-behaviour evaluation regresses.
• Track guardrail trips against an established baseline.
• Alert when a scheduled AI job produces no output.

User-reported degradation

Give users a one-click way to flag a bad response, route reports to an owner, and treat a cluster as an incident signal.

User reports can detect answer-quality failures that return successful HTTP responses. Use them alongside evaluations and canaries because infrastructure metrics may remain stable.