7.4 Runbooks

A runbook is a step-by-step procedure for one situation, written in calm conditions so that it can be followed under pressure by someone who did not write it. When something breaks at night, nobody should have to remember or improvise how to restore a service.

Runbooks answer "how do I do this?". Deciding what is happening and who leads the response belongs to Incidents.

7.4.1 Rules

• Every alert links to one runbook. An alert without a runbook is not ready to be enabled.
• One template for all, so an unfamiliar runbook reads like a familiar one.
• Company-wide runbooks live here; a runbook for one project lives in that project's docs/ folder and is linked from here.
• Every runbook has an owner and is walked through by a new team member during onboarding. A step that confuses them is rewritten.
• After every incident, the runbook used is updated with what was missing.
• A step that is always the same and safe becomes a script or a pipeline; the runbook then says when to run it.

7.4.2 Template

# <What this restores or fixes>

When to use: <the alert name, or the symptom someone reports>
Impact: <who is affected and how>
Access needed: <grant or role, and how to get it>

## Check
1. <how to confirm the situation, with the command or Console page>

## Fix
1. <the smallest step that restores service>

## Verify
1. <how to know it worked>

## If it does not work
<who to call, what to try next>

## Afterwards
<what to record, which issue to open>

7.4.3 Runbooks

Runbook	Use it when
Service Recovery	A service in prod fails after a release, or stops answering
Database Recovery	Data in a prod database was deleted or damaged
AI Tool Delivery	How MCP tools move from product capability to authorized, observable execution
Stage Release and Recovery	Release a registered stage component, resume an interrupted run or recover a failed rollout

Planned, to be written before the first prod release: secret rotation, recovery of a Terraform state, and a sandbox that stopped itself on its budget.