A weekly agent release went out at 18:40 on the Friday of a year-end peak. It passed its tests, and the regression it carried was invisible to anyone reading the bill: spend was already up with the season. Twenty-two minutes later Pointee had flagged it, and four minutes after that the agent was back on the previous version — $2,130 spent instead of $427,000.

The situation

Year-end is when the bank's document volumes peak and when releases are frozen for everything except the agents, which ship weekly. Version 14 of the document-verification agent passed its tests and was released at 18:40 on a Friday. The regression it carried was invisible to a human reader of the bill: total spend was already up with the season, and the cost dashboard showed totals, not cost per case against a baseline.

What the team did with Pointee

18:40 Release. Within five cases, credits per case jumped from a 7-day baseline of 38 to a running mean of 340.
19:02 Detect. The Guardian Agent's rule triggered after 47 cases. An incident was opened with the per-case chart attached.
19:06 Contain. Automatically rolled back to v13; the owner and the on-duty engineer were notified. In-flight cases finished on v13. Overspend stopped.
Tue Fix, safely. v15 shipped behind a 5% canary and a 60-credit cap per case. All 47 v14 cases were re-verified. The incident was recorded in 3 hours.

The incident record

Incident recordWithoutWith PointeeChange
Time to detectMonday, if ever22 min−61 hours
Time to containhours4 minautomatic
Overspend$427,000$2,130−$424,870
Half-finished casesunknown0done on v13
Evidence for the auditin days3 hfirst-party
Credits per case · Friday 17:00–21:00
A 7-day baseline of 38 credits per case, the v14 spike to a running mean of 340, and the return to baseline after rollback.
400 300 200 100 0 17:00 18:00 19:00 20:00 21:00 Baseline 38 credits / case 351 18:40 — v14 released, baseline 38 credits per case 19:02 — Guardian Agent flags 9.1× baseline, 351 credits per case 19:06 — rolled back to v13, back to baseline 18:40 · v14 released 19:02 · Guardian Agent flags 9.1× baseline 19:06 · rolled back to v13, cases continue

Why it was caught at all

  • Cost per case, against a baseline. Totals hid the regression inside a seasonal rise; credits per case against a 7-day baseline made a 9.1× jump obvious within 47 cases.
  • Containment did not wait for a human. The rollback ran automatically at 19:06 on a Friday evening — the notification followed it rather than gating it.
  • No case was left half-finished. In-flight work completed on v13, so the rollback cost nothing in rework or manual clean-up.
  • The evidence was already written. The incident record took 3 hours to assemble from first-party data, not days of reconstruction.

Conclusion

Nothing failed. The agent passed its tests, every case completed, and the bill was rising for reasons the season explained. What caught the regression was watching cost per case against its own baseline rather than watching totals — and acting on it in four minutes without waiting for a Friday-evening escalation.