Useful signals
Signals tied to operating decisions, including access for application owners and audit teams.
An operations dashboard should support a named decision and a known owner. Panels that do neither can remain available as queries for occasional investigation.
Decision-linked metrics
A metric belongs on the main dashboard when a team can name the decision it informs.
Aggregate token counts may provide investigation context. Cost per session, scoped to an application owner, can support a specific optimization or budget decision.
For each panel, record the decision and owner. Move panels with no current use into saved queries.
Instrumentation goals
Useful AI observability supports these goals:
- Current user experience, measured through task success rate, error rate, and time-to-first-token.
- Quality trend, measured through evaluation scores, refusal rate, tool calls per task, and user reports.
- Action reconstruction through traces.
- Cost attribution for finance and application owners.
Keep supporting context available without placing every measure on the main dashboard.
Leading and lagging
Error rate is lagging: by the time it moves, users are affected. The useful work is finding indicators that move first.
Useful leading indicators include:
- Rising tool calls per completed task can show degraded retrieval or instructions.
- Rising retry rate can show a downstream error or more output-validation failures.
- Growing prompt length can show unplanned context accumulation and will affect cost and latency.
- A falling score on the pinned evaluation suite directly measures a quality regression.
A home-price forecast and a $304 million write-down
Zillow shut down Zillow Offers, its algorithmic home-buying (iBuying) business, on November 2, 2021, after its home-valuation models systematically overpaid for houses as US price growth decelerated. Q3 2021 results carried a $304 million inventory write-down, with a further $240 million to $265 million of losses expected in Q4, and roughly 25% of the workforce was cut.
The skipped letter. Observe was the missing letter: the ground truth for the model's error, the spread between what Zillow paid and what its homes resold for, accrued in Zillow's own transaction data for months, yet the scale of the miss surfaced through quarterly accounting as a $304 million write-down rather than through model monitoring that tracked forecast error against realized sale prices in a turning market. Control failed second: when the error did surface, the only available levers were blunt, a full purchase freeze on October 18 and shutdown of the entire business two weeks later; nothing graduated tied offer aggressiveness back to observed pricing error while the position was still recoverable.
Zillow Group Q3 2021 results and plan to wind down Zillow Offers (press release, Nov 2, 2021) / Zillow Group Form 8-K exhibit 99.1, Q3 2021 (SEC EDGAR) / Zillow stock plunges after company exits home-buying business (CNBC, Nov 3, 2021)
Access for owners and auditors
Give analysts, workflow owners, and compliance staff access to the records required for their roles.
Builders need application cost, errors, recent traces, and evaluation results for the systems they can manage. Access should not require a query language.
Workflow owners need task success rate, volume, cost trend, and user-reported problems.
Audit staff need a record for a date range and system, including activity, data access, and applicable approvals from compliance controls.
Provide each view from the same underlying telemetry so decisions use consistent data.
Questions to ask your vendor, or your own team
- For each panel on your main AI dashboard: what decision does it change, and whose?
- Which metrics provide a leading indication of failure?
- Can the person who built an application see its cost and traces themselves?
- Is there a view for a non-technical person who owns the outcome?
- When was your dashboard last opened during an actual incident?
- What is the setpoint for your most-watched metric?