Secrets

Secrets management for AI workloads is the existing discipline with two new leaks: the prompt and the trace. Everything else you already know how to do.

The old rules still apply

No secrets in source. No secrets in container images. No long-lived credentials where a short-lived one would work. Injection at runtime from a secret store, scoped to the workload identity. Rotation that does not require a change window.

If your AI applications do not meet that bar, fix that first. It is not an AI problem and there is nothing novel about it.

The two new leaks

The prompt. People paste credentials into prompts. Not out of carelessness, usually: they are debugging a failing API call and they paste the whole request, header included. That prompt is now in the provider’s logs, your prompt log, the conversation history table, and possibly a trace. One paste, five copies.

The trace. An agent trace captures tool calls and their arguments, which is exactly what makes it useful and exactly what makes it dangerous. A tool call that authenticates will put the credential in the trace unless something redacts it.

Both need the same treatment: detection and redaction on the way in, not a policy telling people not to do it. Pattern-match for known credential shapes at the gateway and at the trace collector, redact before storage, and alert when it happens so you can go fix the workflow that made it necessary.

Agents hold credentials for a long time

A web request holds a credential for milliseconds. An agent session holds one for the length of the session, in a process that is also executing generated code and reading untrusted content. That is a longer window and a worse neighbourhood.

Three mitigations, in order of value:

1. Scope down. The credential the agent holds should permit exactly the operations the task needs, not the union of everything the team can do.
2. Shorten. Issue at session start, expire at session end. If a session outlives its credential, it should re-request rather than hold a long one.
3. Broker instead of hand over. The strongest pattern: the agent never holds a credential at all. It calls a broker that performs the privileged operation and returns the result. Then a compromised or confused agent can invoke an operation but cannot exfiltrate the key.

The third is more work and it is the one worth building for anything touching production.

Chat history is a secret store you did not design

If your organization uses a chat interface to a model, assume credentials are in it. Assume the same for ticket comments, incident channels, and pull request descriptions, all of which now feed retrieval indexes at many companies.

This is worth stating plainly because the mitigation is unglamorous: scan what you index, scan what you log, and rotate on detection. The alternative is to discover it during an audit, at which point the question is not whether it happened but for how long.