Routing

Routing is the decision, made at request time, about which model or provider serves a call, and whether that decision can be changed centrally.

Central routing lets an application change providers without code changes. It also lets the same application use a vendor endpoint in one placement and a local model in another, which supports portability. Routing rules can enforce which endpoints are contractually approved for sovereign workloads.

Direct provider calls

Application code names a provider and a model. It reads a key from an environment variable and calls the provider SDK directly.

Each application then needs a code change and deployment to switch models. Provider invoices become the source for cost attribution, and cross-cutting controls must be added separately to every application. A new restriction on provider use also becomes an application project that requires changes across the affected codebases.

Routing-layer decisions

A gateway can choose a model by application, environment, data classification, or caller. For regulated data, it should route tagged requests only to covered endpoints.

The gateway should select a pinned model version and apply bounded retries. It may fail over to another provider or use a smaller model when a request fails. It is also a central enforcement point for redaction and other policy checks.

Record token usage and cost for every call. Include latency and the selected model, along with the caller's identity. These records feed metrics.

Pin by default

If an application requests a model by name without a version, the provider controls when its behaviour changes. Users may encounter the change before the platform team does.

Pin the version in the routing layer. Take updates as a deliberate change: run the eval suite against the new version, compare, then move the pin. The version becomes configuration, which means it is reviewable and revertible like any other change.

Keep a floating alias alongside the pinned one for a canary that runs the same suite continuously. When the two diverge, you learn about a provider change on your own schedule.

Test failover behaviour

Providers with the same interface can produce different behaviour. A prompt tuned for one may perform worse on another, so failover can introduce a quality regression.

If failover matters, run the eval suite against the fallback too, and know what you are falling back to. Tell users in the response or interface when the fallback performs meaningfully worse.

A smaller model may be an appropriate fallback for a tested subset of requests. Return a clear message for requests outside that subset. Do not send those requests to an untested model.