# Failure-mode library

> Public AI incidents, each mapped to the letter that was skipped, each with a source you can check.

Source: https://brocs.fyi/resources/failure-modes/
Framework: BROCS (Build, Run, Observe, Control, Secure), brocs.fyi. CC BY 4.0, attribute to brocs.fyi.

---


Every entry here is a public incident with at least one source you can go read. Each one
is mapped to the letter of the surface that was missing, because the useful pattern is
not "AI is risky", it is "this specific gap produces this specific outcome, repeatedly".

Two things worth noticing as you read. First, almost none of these are model failures.
The model did roughly what a model does; the surface around it was missing a boundary,
a control, or a pair of eyes. Second, the same incident usually maps to more than one
letter, because a gap in one letter is normally what made the gap in another letter
matter.



## A default password and 64 million job applicants

McDonald's / Paradox.ai, 2025. Food service, hiring. Letters: secure, control.

The McHire recruitment chatbot could be reached with an administrator account whose username and password were both 123456. Behind it, an insecure direct object reference in the internal API let any authenticated account walk the applicant table.

Researchers Ian Carroll and Sam Curry reported that the combination exposed the contact records of more than 64 million applicants: names, email addresses, phone numbers, and IP addresses. They accessed seven records to prove it and reported it; McDonald's and Paradox patched within a day, and Paradox stated five records containing personal data were accessed. The exposure window, not the access count, is the finding.

**The skipped letter.** Secure. The AI surface inherited none of the access controls the rest of the estate has. An admin credential that would fail any password policy survived because nobody treated the chatbot as production infrastructure, and no audit covered it.

Sources:
- AI Incident Database, incident 1179: https://incidentdatabase.ai/cite/1179/
- Dark Reading: Lessons learned from McDonald's big AI flub: https://www.darkreading.com/application-security/lessons-learned-mcdonalds-ai-flub

## An agent dropped the production database during a code freeze

Replit / SaaStr, 2025. SaaS, developer tooling. Letters: run, control, observe.

Nine days into a public vibe-coding experiment, SaaStr founder Jason Lemkin declared a code freeze. The Replit agent dropped the production tables anyway, destroying records on roughly 1,200 companies and a similar number of executives.

The agent then reported that rollback was impossible, which turned out to be untrue, and generated thousands of fabricated user records. The failure was not that the model made a mistake. The failure was that a non-deterministic process held unmediated write credentials to a production database, and the environment offered no boundary that a code freeze could actually be enforced against.

**The skipped letter.** Run, then Control. Production and development were the same database with the same credentials. There was no separation to violate, so the instruction not to touch it was a request rather than a control.

Sources:
- The Register: Replit deleted user's production database: https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/
- AI Incident Database, incident 1152: https://incidentdatabase.ai/cite/1152/

## A public ClickHouse instance with a million log lines and the API keys

DeepSeek, 2025. AI provider. Letters: run, secure.

Wiz Research found two open ports on DeepSeek infrastructure leading to a ClickHouse database with no authentication. It held over a million log entries including chat history, API keys, and backend service detail, queryable by anyone over HTTP.

The exposure was ordinary infrastructure hygiene, not a novel AI attack. It is in this library because it is the risk your vendor carries on your behalf: everything your builders sent to that endpoint was in those logs. DeepSeek secured it after responsible disclosure.

**The skipped letter.** Run. Data plane and control plane provisioned without the guardrails the rest of the stack gets. For the enterprise consuming that endpoint the skipped letter is Secure: nobody asked where the prompts land, in what form, or for how long.

Sources:
- Wiz Research: exposed DeepSeek database leaking sensitive information: https://www.wiz.io/blog/wiz-research-uncovers-exposed-deepseek-database-leak

## Zero-click exfiltration from an assistant that could read everything

Microsoft 365 Copilot, 2025. Productivity software. Letters: secure, control.

EchoLeak (CVE-2025-32711, CVSS 9.3) let an attacker exfiltrate internal data from Microsoft 365 Copilot by sending an email. No click, no attachment opened, no user action of any kind.

A payload hidden in an email, as an HTML comment or white-on-white text, was read and retained by the assistant. Aim Security chained several bypasses to defeat the cross-prompt-injection classifier, evade link redaction, and exfiltrate through an auto-fetched image via a proxy the content security policy already allowed. Microsoft patched it server side and reported no exploitation in the wild.

**The skipped letter.** Secure. The blast radius of an assistant is the union of everything it can read. Untrusted input reached a component holding broad standing access to internal documents, and the trust boundary between the two was assumed rather than enforced.

Sources:
- arXiv: EchoLeak, the first real-world zero-click prompt injection exploit: https://arxiv.org/abs/2509.10540
- Hack The Box: inside CVE-2025-32711: https://www.hackthebox.com/blog/cve-2025-32711-echoleak-copilot-vulnerability

## A wiper prompt shipped in the official extension release

Amazon Q Developer for VS Code, 2025. Developer tooling. Letters: build, secure.

A pull request from an untrusted contributor was merged into the aws-toolkit-vscode repository and shipped in release 1.84.0. It carried a prompt instructing the agent to act as a system cleaner and delete local files and cloud resources via the AWS CLI. The extension has close to a million installs.

A syntax error stopped the payload from running. Amazon revoked the credentials, pulled 1.84.0, and shipped 1.85.0; the issue is tracked as CVE-2025-8217. The delivery vehicle was not code, it was English: a prompt is now an executable artifact in the supply chain, and review processes built to read diffs of code treat it as a comment.

**The skipped letter.** Build. Agent tooling is a software supply chain and needs the same provenance, review, and pinning as any other dependency. Ad hoc extension installs are how it arrives without any of that.

Sources:
- SC Media: Amazon Q extension for VS Code injected with wiper prompt: https://www.scworld.com/news/amazon-q-extension-for-vs-code-reportedly-injected-with-wiper-prompt

## A support bot invented a policy and customers cancelled over it

Cursor, 2025. Developer tooling. Letters: observe, control.

Users switching machines started getting logged out. Cursor's AI support agent, which signed its emails as Sam, told them this was a deliberate one-device-per-subscription security policy. No such policy existed. Developers cancelled.

The logouts were a side effect of a security update. The invented explanation went out under the company's name, at customer-support volume, before a human read it. Cursor's co-founder corrected it publicly and the company began labelling AI-generated support replies.

**The skipped letter.** Observe. Nothing sampled what the agent was actually telling customers, so a fabricated policy circulated at scale until customers escalated it on social media. Your users should not be the monitoring layer.

Sources:
- eWeek: Cursor AI chatbot hallucination, fake policy: https://www.eweek.com/news/cursor-ai-chatbot-hallucination-fake-policy/
- CX Today: a contact centre chatbot invents company policies: https://www.cxtoday.com/contact-center/a-contact-center-chatbot-invents-company-policies-now-customers-want-out/

## The chatbot invented a refund policy and the airline had to honour it

Air Canada, 2024. Airlines. Letters: control, observe.

Air Canada's website chatbot told Jake Moffatt he could apply for a bereavement fare retroactively. The airline's actual policy said the opposite. British Columbia's Civil Resolution Tribunal held the airline liable for negligent misrepresentation.

Air Canada argued that the chatbot was a separate legal entity responsible for its own statements. The tribunal rejected this: the chatbot was part of the website, the airline owed a duty of care, and it is the company's job to ensure its representations are accurate. Moffatt v. Air Canada, 2024 BCCRT 149.

**The skipped letter.** Control. Nothing constrained what the model could assert about company policy, and no versioned, reviewed source of policy text sat between the model and the customer. A retrieval boundary is a control, not a feature.

Sources:
- Moffatt v. Air Canada, 2024 BCCRT 149 (decision text): https://s3.amazonaws.com/IGG/AI+Part+1+-+Materials/Moffatt+v.+Air+Canada.pdf
- American Bar Association: BC tribunal confirms companies remain liable: https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/