Sample content: This article exists to exercise the publishing system. It is not presented as Patrick’s writing.
Give Experiments a Failure Budget
A sample management framework for deciding how much reliability, time, and user trust an experiment is allowed to risk.
“Move fast” is incomplete guidance. Fast in a prototype, a billing system, and a migration of customer data describe different risk contracts. A failure budget makes that difference discussable.
The phrase is not permission to be careless. It is an agreement about what may fail, who may be affected, how the team will know, and how quickly the experiment can be stopped. Clear limits make experimentation safer and often faster because fewer risks need to be renegotiated midstream.
Budget more than uptime
Copy link to section “Budget more than uptime”An experiment can spend several forms of trust:
- customer time;
- data integrity;
- operational attention;
- team focus;
- reversibility;
- public credibility.
These budgets interact. A read-only internal prototype may tolerate frequent errors but no exposure of customer data. A small customer beta may tolerate a rough workflow but require reliable rollback and responsive support. “Low traffic” does not automatically mean “low consequence.”
Name the protected invariant. Data must remain recoverable. Existing workflows must continue to work. No customer should be charged. Support staff must be able to identify participants. Specific invariants lead to specific controls.
Put the guardrail beside the goal
Copy link to section “Put the guardrail beside the goal”The experiment brief should state both what it hopes to learn and what it is not allowed to damage. A feature flag, limited cohort, read-only path, synthetic data set, or manual rollback can convert a vague appetite for risk into an operating plan.
Put the control on the same path as the risk. A rollback document nobody can reach during an incident is weaker than a verified switch. A cohort limit in a planning note is weaker than a technical constraint that prevents accidental expansion.
A concise failure-budget checklist can ask:
- Who or what is exposed?
- Which failures are acceptable during learning?
- Which outcome stops the experiment immediately?
- How will the team detect that outcome?
- Who has authority and ability to stop it?
- What cleanup remains if the idea is abandoned?
A useful failure budget does not guarantee success. It makes an unsuccessful result survivable and informative.
Revisit the budget as evidence changes
Copy link to section “Revisit the budget as evidence changes”Early uncertainty may justify a tiny cohort. Repeated safe operation may justify expansion. A near miss should narrow the budget until the team understands the failure mode.
Expansion should follow evidence, not elapsed time. A test that ran quietly because nobody used the risky path has not demonstrated reliability. Look for exposure to the conditions the next stage will introduce.
Make cleanup part of the budget
Copy link to section “Make cleanup part of the budget”Experiments leave flags, data, dashboards, temporary access, and code paths. If cleanup has no owner or deadline, the organization accumulates permanent complexity from temporary learning.
Define the end states before launch: graduate the capability into supported production, revise it for another bounded trial, or remove it completely. A failure budget protects the future system as well as today’s users.
Good experimentation does not eliminate failure. It turns failure into affordable information while keeping trust outside the blast radius.
