# Routing

> Model and provider routing, pinning, and failover: how switching cost is held near zero.

Source: https://brocs.fyi/control/routing/
Part of the surface: C · Control. Keep a hand on the wheel.
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---

## The Control surface

- Configuration: declarative, versioned, reviewed, reversible
- Ingress: one front door for internal AI apps
- Identity: SSO by default, external access without VPN sprawl
- Storage and backups: retention, lifecycle, restore drills
- Routing: model and provider routing, pinning, failover
- Remote access: work reaches the data instead of the data reaching the laptop


Routing is the decision, made at request time, about which model or provider serves a
call, and whether that decision can be changed centrally.

It is the mechanism that makes several other claims in the framework true. Tool churn is
survivable because routing absorbs it. [Portability](/run/portability/) is possible
because the same application can use a vendor endpoint in one placement and a local model
in another. [Sovereignty](/resources/glossary/#sovereignty) is achievable because the
routing layer knows which endpoints are contractually covered and enforces it.

## The anti-pattern

Application code names a provider and a model. It reads a key from an environment
variable and calls the provider SDK directly.

Everything downstream inherits that decision. Changing model means a code change and a
deploy, per application. Cost attribution has to be reconstructed from provider invoices.
There is no single place to add a guardrail, a rate limit, or a redaction step. And when
a regulated business unit says their data cannot go to that provider, the answer is a
project rather than a configuration change.

## What the routing layer decides

A gateway sitting between applications and providers can make several decisions
consistently, and each one is painful to retrofit:

**Which model.** By application, by environment, by data classification, or by the caller.
The classification case is the one that matters most in regulated organizations: requests
tagged as containing protected data route only to covered endpoints, and that is enforced
rather than documented.

**Which version.** Pinned by default. See below.

**What happens on failure.** Retry with a bound, fail over to a second provider, or
degrade to a smaller model. Decided once, applied everywhere, and bounded so retries
cannot become a budget event.

**What is allowed through.** Redaction of credential patterns, personal data handling,
and whatever policy checks apply, in one place rather than in each application.

**What gets recorded.** Tokens, cost, latency, model, and identity, on every call. This is
where [metrics](/observe/metrics/) come from.

## Pin by default

If your application requests a model by name without a version, the provider decides when
your behaviour changes, and you find out from a user.

Pin the version in the routing layer. Take updates as a deliberate change: run the
[eval suite](/observe/model-behavior/) against the new version, compare, then move the
pin. The version becomes configuration, which means it is reviewable and revertible like
any other change.

Keep a floating alias alongside the pinned one for a canary that runs the same suite
continuously. When the two diverge, you learn about a provider change on your own schedule.

## Failover is not free

Two providers with the same interface is not two providers with the same behaviour. A
prompt tuned for one will perform differently on the other, sometimes much worse, and a
failover that silently degrades quality is a quiet incident rather than a save.

If failover matters, run the eval suite against the fallback too, and know what you are
falling back to. If the fallback is meaningfully worse, say so in the response or in the
interface rather than pretending nothing happened.

The honest version of failover for many organizations is a smaller, cheaper, faster model
that handles a subset of requests correctly, plus a clear message for the rest. That is
more useful than a second frontier model nobody has tested.

{{< aside title="How API gateways think about this" >}}
Everything on this page is the standard API gateway argument, applied to model providers:
centralize cross-cutting concerns so applications do not each implement them differently
and wrongly. The reason it usually does not happen here is that calling a provider SDK
directly is three lines and works immediately, so the first ten applications do it that
way before anyone builds the gateway. Retrofitting is possible and it is a lot of small
pull requests.
{{< /aside >}}

{{< checklist >}}
- Could you move one application to a different model provider without a code change?
- Is the model version pinned, and who decides when it moves?
- Does anything enforce that classified data only reaches approved endpoints?
- What is your retry policy, and what bounds it?
- If your primary provider had a two-hour outage, what happens? Has that been tested?
- Where is the single record of every inference call your organization makes?
{{< /checklist >}}