Keys and models

A model API key is a bearer credential with a billing account attached and, in most organizations, no expiry. It is the most casually handled secret in the building.

The spreadsheet is the incident

The common pattern goes like this. One team needs model access. Someone with a corporate card creates a provider account and generates a key. The key goes into a shared document so the rest of the team can use it. Six months later that document has been copied into two other teams, the key is in four repositories and a CI system, the person who created it has changed roles, and nobody can say what would break if it were rotated.

That is not a security lapse by an individual. It is the predictable output of having no key distribution system, and the fix is not a training module.

The fix is that keys are issued by something, scoped to something, and expire. Concretely:

• Nobody holds a provider key. Applications and tools authenticate to an internal gateway with their own identity. The gateway holds the provider credential.
• Every call is attributable. The gateway knows which app, which team, and which user made the request, because that is the only way cost and traces work later.
• Rotation is a routine operation. If rotating the provider key requires a change window and a list of teams to notify, it will not happen. Behind a gateway it is one change in one place.
• Revocation is per-consumer. Cutting off one misbehaving app should not take down everything else that shares the key.

Four ways to get model access, and why you need more than one

Vendor SaaS endpoint. Fastest to adopt, least control. Fine for a large class of internal work, and unacceptable for another class. The decision belongs to a policy, not to whichever engineer moved first.

BYOK. Your provider credentials, your account, your billing, your data-processing terms, someone else’s application. This is the pattern that lets a regulated organization use a third-party product without the product vendor becoming a processor of the underlying data.

Models through your own cloud. The same model families served inside your existing cloud account, under the contract, network boundary, and audit regime you already have. For many enterprises this is the shortest path from “we cannot use that” to “we can use that”, because it does not require a new vendor relationship at all.

Local or self-hosted models. Slower and more work, and the only option when data genuinely cannot leave a boundary. Also the only option that is immune to a provider deprecating a model you depend on.

Regulated work adds a fifth consideration: BAA-covered endpoints. If protected health information touches the model, the endpoint has to be covered by a Business Associate Agreement. Whether an endpoint is covered is a contractual fact, not a technical one, and it needs to be recorded somewhere the routing layer can read.

Most organizations end up with three of these at once, which is exactly why the routing decision has to be centralized rather than made independently in each application. That is a Control concern, and it is the payoff for doing keys properly.

Metering is not billing

Metering is knowing who spent what. Billing is deciding who pays. You need the first long before you need the second.

Without metering, a surprising invoice produces a blunt response: a spend cap that stops everyone, or a policy that stops the experimentation you were trying to encourage. With metering, the response is proportionate, because you can see that one batch job is 80 percent of the bill.

Meter at the gateway, on every call: model, tokens in and out, latency, app, team, user. Everything in cost depends on that record existing.

What a key in the wrong place actually costs

The risk is not only unauthorized spend. A leaked key is an authenticated path into your provider account, and provider accounts increasingly hold more than inference: fine-tuned models, uploaded files, assistant configurations, and logs of everything anyone sent.