S · Secure

Governance

Policy encoded where it is enforced, not written where it is filed.

View as Markdown

Governance is the set of rules about what your organization will and will not do with AI, and the mechanisms that make those rules true. The second half is the part that usually does not exist.

The document is not the control

Most organizations write an AI policy early. It says sensible things: do not put customer data in unapproved tools, use approved models, get review before customer-facing deployment.

Then someone builds something. They do not read the policy, not out of defiance, but because it is in a wiki they have never opened and they are trying to solve a problem today. Nothing in the path they took mentions the policy, and nothing checks.

The policy is now a statement about intent. It is not a control, and treating it as one is how organizations end up genuinely surprised by their own systems.

The test: for each rule in your AI policy, name the mechanism that enforces it. Rules with no mechanism are aspirations, which is fine as long as everyone knows which ones those are.

Encode it where the action happens

Each rule has a natural enforcement point, and it is usually somewhere in a letter of the surface rather than in a governance system.

PolicyEnforcement point
Only approved modelsThe routing layer, by classification
No customer data to unapproved endpointsRouting plus redaction at the gateway
Customer-facing deployments get reviewIngress: public exposure is a gate
Agents cannot write to production without approvalTool permissions and a broker, in Run
Retention limitsLifecycle policy on the store, in Control
Spend limitsQuota at the gateway, in cost

None of those live in a governance tool. That is the point: governance is a property of the platform, expressed in the places where things actually happen. A governance tool is useful for recording what the rules are and demonstrating they were followed. It cannot make them true.

Decide the small number of things that need deciding

Governance discussions expand indefinitely if nobody bounds them. The decisions that actually block work are few:

  1. Which data classifications may reach which endpoint types. The single most consequential decision, and the one that unblocks the most work once made.
  2. What requires human approval before it executes. Should be a short list.
  3. What may be customer-facing, and who signs.
  4. What autonomy an agent may have, expressed as tool permissions rather than adjectives.
  5. What must be retained, and for how long.

Five decisions, written down, with enforcement points named. That is a working AI governance position, and it fits on a page.

Approval fatigue is a real failure mode

A human-in-the-loop control is useful in proportion to how rarely it fires. An approval prompt on every action is trained away within a week: people click through it without reading, and you have added latency while removing nothing.

Reserve approvals for actions that are irreversible, externally visible, or expensive. Everything else should be permitted and recorded, with the review happening after the fact on a sample.

Failure mode SC

A default password and 64 million job applicants

McDonald's / Paradox.ai · 2025

The McHire recruitment chatbot could be reached with an administrator account whose username and password were both 123456. Behind it, an insecure direct object reference in the internal API let any authenticated account walk the applicant table.

The skipped letter. Secure. The AI surface inherited none of the access controls the rest of the estate has. An admin credential that would fail any password policy survived because nobody treated the chatbot as production infrastructure, and no audit covered it.

AI Incident Database, incident 1179 · Dark Reading: Lessons learned from McDonald's big AI flub

Questions to ask your vendor, or your own team

  • For each rule in your AI policy, name the mechanism that enforces it.
  • Which data classifications may reach which endpoints, and what checks?
  • What actions require human approval, and how often does that approval get clicked through without reading?
  • Who signs off on a customer-facing AI deployment, and what do they see?
  • If an engineer wanted to break the policy today, what would stop them?
  • When was the policy last changed in response to something you learned?