Sample content: This article exists to exercise the publishing system. It is not presented as Patrick’s writing.
Good Incident Reviews Change the System
A sample reliability essay about turning incident evidence into stronger controls, clearer ownership, and safer operating conditions without blame.
An incident review is not complete when the timeline is accurate or the document is published. Its value appears later, when the system is easier to operate, a dangerous condition is removed, or a team can detect and recover from the next failure sooner.
Review should convert expensive experience into durable change.
Build the timeline from evidence
Copy link to section “Build the timeline from evidence”Start with what can be observed: alerts, logs, deploys, support reports, commands, decisions, and changes in user impact. Separate timestamps that are measured from times reconstructed through memory.
The timeline should explain what information people had at each moment. A decision that looks strange after the root cause is known may have been reasonable with the signals available during the incident.
Avoid assigning intention to gaps in the record. “The engineer ignored the alert” is a conclusion. “The alert fired, was routed to an unstaffed channel, and was not acknowledged” describes a system condition the organization can change.
Look for conditions, not a single culprit
Copy link to section “Look for conditions, not a single culprit”Complex failures rarely require one mistake. They emerge when normal actions meet hidden coupling, weak feedback, unusual load, incomplete safeguards, or assumptions that held until that day.
Ask questions at several layers:
- What changed in the technical system?
- Why did existing controls allow the change to cause impact?
- Why was detection early, late, or misleading?
- Which operating conditions shaped the response?
- What made the chosen action seem safe at the time?
- Which recovery steps depended on rare knowledge or access?
“Human error” stops the investigation at the layer most likely to recur. People will continue to misunderstand, mistype, forget, and act under uncertainty. Resilient systems account for that reality.
Choose actions that alter the failure path
Copy link to section “Choose actions that alter the failure path”An action item should connect to a contributing condition. “Be more careful” and “improve monitoring” are aspirations. Stronger actions identify the control, owner, and expected effect.
| Weak follow-up | Stronger system change |
|---|---|
| Remind deployers to check capacity | Block rollout when verified headroom is absent |
| Add more alerts | Alert on the earliest user-impacting signal with an owned response |
| Update documentation | Put the required check in the deployment path and document why |
| Train everyone on recovery | Automate the safe routine steps and exercise the remainder |
Not every incident justifies automation. The response should be proportional to recurrence, consequence, and the cost of the control. A small documentation fix can be right when the condition is rare and the procedure is inherently judgment-heavy.
Preserve candor through fairness
Copy link to section “Preserve candor through fairness”Blamelessness does not mean decisions cannot be examined. It means the review seeks to understand and improve rather than punish someone for revealing how the system actually behaves.
Leaders shape candor through what happens after disclosure. If the person who describes a near miss becomes the person blamed for it, future near misses will remain hidden. Accountability should focus on owning follow-through and maintaining professional standards, not finding a convenient endpoint for causality.
Follow actions to completion
Copy link to section “Follow actions to completion”Incident work competes with planned delivery as soon as service is restored. Assign owners and dates, but also review whether each action still addresses the condition after implementation details emerge.
Close the loop by checking the new control. Run the recovery procedure. Trigger the alert safely. Confirm the dashboard answers the question responders needed. A merged change is not evidence that the operating system improved.
Good reviews respect the cost users and teams already paid. They leave behind more than a narrative: better feedback, safer boundaries, clearer decisions, and a system that learned.
