Configuration
Declarative, versioned, reviewed configuration for applications, prompts, models, and routing, with tested rollback.
Configuration covers changes to system behaviour that do not require application code. It includes prompts, model selection, routing rules, tool definitions, retrieval settings, guardrail policies, and feature flags.
An AI stack may split this material across an admin database, deployment-time environment variables, and application code. That makes the current state difficult to identify or restore.
Prompts belong in configuration
A prompt can change system behaviour substantially. Direct production edits in an admin panel bypass the normal delivery record.
Manage prompts through the delivery process:
- Store each prompt in version control with a diff and an author.
- Require another person to review customer-facing changes.
- Test changes against the eval suite before release.
- Deploy prompts through a pipeline that supports rollback.
- Record the prompt version with each response at runtime.
Non-engineers may still need to edit prompts quickly. An editor can commit changes to a repository and open a pull request without exposing a production text box that has no history.
Take the tips, serve the cheese
New York City launched the MyCity Business chatbot in October 2023, powered by Microsoft Azure AI, to answer small-business questions about city rules. In March and April 2024, The Markup (with The City and Documented) and the Associated Press documented it telling employers they could take workers' tips and fire a worker for complaining about sexual harassment, telling landlords they could refuse housing-voucher tenants, and telling a restaurant it could still serve cheese nibbled by a rat. The city acknowledged the wrong answers and left the bot online.
The skipped letter. Control was the skipped letter: the bot sat on the city's official domain answering questions of law, yet nothing in its configuration bound it to the city's own statutes and guidance; there was no retrieval boundary and no output constraint stopping an Azure model from confidently inverting NYC labor, housing, and consumer law under a government masthead. Observe was skipped alongside it: nobody at the city was sampling outputs, so journalists found the failures first, the city's only counterclaim was unaudited ("thousands of timely, accurate answers"), and the post-publication remedy was a disclaimer rather than measurement, after which the bot ran wrong for nearly two more years.
NYC's AI Chatbot Tells Businesses to Break the Law (The Markup, Mar 29, 2024) / Malfunctioning NYC AI Chatbot Still Active Despite Widespread Evidence It's Encouraging Illegal Behavior (The Markup/The City, Apr 2, 2024) / NYC's AI chatbot was caught telling businesses to break the law. The city isn't taking it down (AP, Jake Offenhartz, Apr 3, 2024, via NBC New York)
Declare the desired state
Describe the intended state in a source of truth, then have the system apply it. Changes made by hand through a console leave the current state only in the running system and make it difficult to reproduce.
AI configuration changes frequently and includes security-relevant choices. Model assignments and agent tool permissions should be reviewable. The same applies to retrieval access and guardrail policies.
Test rollback
Rollback should restore a known working state quickly and leave a record of the change.
Pick a prompt change from last month and time its production rollback. A process that depends on finding the author and asking what the prompt used to say is too slow and relies on memory.
A prompt rollback may leave a retrieval index built for the newer prompt. That combination may never have run before, so index changes need their own rollback plan.
Configuration rollback also cannot restore a provider model that moved underneath a floating alias. It restores only the configuration you control. See routing.
Make environment differences explicit
Development, staging, and production will use different models, different keys, and different data. Guardrail settings may differ as well. Document those differences so a staging prompt does not reach production and a production key does not end up in a notebook.
Keep the shape of the configuration identical across environments and let only values differ, so a diff between environments is short and readable.
Questions to ask your vendor, or your own team
- Where is the current production prompt, and who changed it last?
- Given a response from yesterday, can you identify the prompt version that produced it?
- How long would it take to roll back a prompt change, and has anyone tried?
- Is model selection configuration, or is it in application code?
- Which tools can each agent call, and where is that list?
- Can you diff staging configuration against production in one command?