# Model behaviour

> Drift, silent provider updates, and quality regressions: the failures that return a 200 and look like nothing on a dashboard.

Source: https://brocs.fyi/observe/model-behavior/
Part of the surface: O · Observe. See what it is doing.
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---

## The Observe surface

- Metrics: golden signals plus tokens, cost, tool calls
- Traces: reconstruct what an agent did, with what, on what data
- Alerts: routed to the builder who owns the app
- What matters: signals that predict incidents, not dashboard decoration
- Model behavior: drift, silent provider updates, quality regressions


Every other page in [Observe](/observe/) is about a system failing in ways your existing
monitoring understands. This one is about the failure that returns 200, completes in
normal time, and gives a worse answer than it did last week.

## Four ways behaviour changes without you changing anything

**The provider updated the model.** You called the same name and got a different model.
Behaviour shifts, sometimes better, sometimes worse for your specific prompts, and the
only notice was a changelog entry.

**The retrieval corpus changed.** Someone updated the source documents, or the index
rebuilt with a different chunking strategy, and the model is now answering from different
material.

**The prompt changed upstream of you.** A shared system prompt, a policy header, or a
tool description was edited by another team. Your application inherited it.

**The input distribution changed.** Nothing in your system moved. Users started asking
different questions, or the same questions differently, and the system was never good at
those.

Only the fourth is visible without deliberate measurement, and only sometimes.

## Evals are the answer, and they are unglamorous

An eval is a fixed set of inputs with a way of judging output. Run it on a schedule and
on every change. That is the whole idea.

What makes an eval suite useful:

**It contains your actual failures.** The highest-value cases are the ones that went
wrong in production. Every incident should end with a case added.

**It is versioned with the thing it tests.** A suite that drifts alongside the prompt
cannot detect regression, because both moved.

**It is small enough to run constantly.** Fifty good cases you run on every change beat a
thousand you run quarterly.

**It scores something specific.** "Is this good" is not a scoring function. Did it cite
the right document, did it produce valid JSON, did it refuse when it should have, did it
avoid the phrase legal asked about.

**It has a pinned control.** Run the same suite against a pinned model version alongside
the floating one. When they diverge, you know it was the provider rather than you.

## Pin, then move deliberately

If your application names a model without a version, the provider decides when your
behaviour changes.

Pin the version. Take the update as a change: run the eval suite against the new version,
compare, then move. That converts a surprise into a scheduled task, and it makes the
question "did anything change" answerable.

The counterargument is that pinning means missing improvements and eventually hitting a
deprecation. Both are true, and both are manageable, which is more than can be said for
discovering a behaviour change through a customer complaint. See
[pinning](/control/routing/).

## The canary that costs almost nothing

Send a small set of known inputs through the production path on a schedule, several times
an hour, and compare against expected properties. Not to grade quality precisely, just to
notice a step change.

It is cheap, it catches provider incidents and configuration mistakes long before users
report them, and it works even when real traffic is too low or too varied to trend.

{{< failure id="air-canada-chatbot" >}}

{{< aside title="How machine learning operations thinks about this" >}}
The MLOps discipline has been dealing with drift for years and arrived at monitoring input
distributions and output distributions separately, because they fail differently. The
amendment for generative systems is that the model itself is now a moving dependency you
do not control, which is closer to a third-party API than to a model you trained. Version
pinning and contract testing are the right analogies. Retraining is not.
{{< /aside >}}

{{< checklist >}}
- Does your application specify a model version, or just a model name?
- Do you have an eval suite? How many cases, and when did it last run?
- Does every production incident end with a new eval case?
- Would you find out about a quality regression from your own measurements or from a user?
- Who else can change a prompt your application depends on?
- What is your canary, and how often does it run?
{{< /checklist >}}