Routing

Routing is the decision, made at request time, about which model or provider serves a call, and whether that decision can be changed centrally.

It is the mechanism that makes several other claims in the framework true. Tool churn is survivable because routing absorbs it. Portability is possible because the same application can use a vendor endpoint in one placement and a local model in another. Sovereignty is achievable because the routing layer knows which endpoints are contractually covered and enforces it.

The anti-pattern

Application code names a provider and a model. It reads a key from an environment variable and calls the provider SDK directly.

Everything downstream inherits that decision. Changing model means a code change and a deploy, per application. Cost attribution has to be reconstructed from provider invoices. There is no single place to add a guardrail, a rate limit, or a redaction step. And when a regulated business unit says their data cannot go to that provider, the answer is a project rather than a configuration change.

What the routing layer decides

A gateway sitting between applications and providers can make several decisions consistently, and each one is painful to retrofit:

Which model. By application, by environment, by data classification, or by the caller. The classification case is the one that matters most in regulated organizations: requests tagged as containing protected data route only to covered endpoints, and that is enforced rather than documented.

Which version. Pinned by default. See below.

What happens on failure. Retry with a bound, fail over to a second provider, or degrade to a smaller model. Decided once, applied everywhere, and bounded so retries cannot become a budget event.

What is allowed through. Redaction of credential patterns, personal data handling, and whatever policy checks apply, in one place rather than in each application.

What gets recorded. Tokens, cost, latency, model, and identity, on every call. This is where metrics come from.

Pin by default

If your application requests a model by name without a version, the provider decides when your behaviour changes, and you find out from a user.

Pin the version in the routing layer. Take updates as a deliberate change: run the eval suite against the new version, compare, then move the pin. The version becomes configuration, which means it is reviewable and revertible like any other change.

Keep a floating alias alongside the pinned one for a canary that runs the same suite continuously. When the two diverge, you learn about a provider change on your own schedule.

Failover is not free

Two providers with the same interface is not two providers with the same behaviour. A prompt tuned for one will perform differently on the other, sometimes much worse, and a failover that silently degrades quality is a quiet incident rather than a save.

If failover matters, run the eval suite against the fallback too, and know what you are falling back to. If the fallback is meaningfully worse, say so in the response or in the interface rather than pretending nothing happened.

The honest version of failover for many organizations is a smaller, cheaper, faster model that handles a subset of requests correctly, plus a clear message for the rest. That is more useful than a second frontier model nobody has tested.