7.4.1 Service Recovery

Use this runbook to bring a prod service back to its last good version. Going back first and fixing afterwards restores customers in minutes, without waiting for a new build.

Production target procedure

Production is not active in the current stage foundation. For the current kit, Access release, failed-job retry and compatible fix/revert path, use Stage Release and Recovery. This production runbook does not prove that a stage operator has rollback permissions or that an older release's migrator is compatible with the current database.

• When to use: errors or slow responses of a prod service right after a release, or a service that stops answering.
• Impact: the users of that service, and the systems that call it.
• Access needed: the Cloud Deploy releaser role on the service's pipeline. Reading logs needs the prod-debug grant (see Identity & Access).

7.4.1.1 Check

1. Open the Moonitoring Console and note the version running in prod and when it was deployed.
2. Compare the start of the errors with the deployment time. If the errors began before the release, a rollback will not help: go to If It Does Not Work.

7.4.1.2 Fix

Roll prod back to the previous release with Cloud Deploy. The rollback is a deployment like any other: it is recorded, and it uses the image that already ran.

gcloud deploy targets rollback prod \
  --delivery-pipeline=<service> \
  --region=europe-west8 \
  --project=moonitoring-cicd

7.4.1.3 Verify

1. The Console shows the previous version in prod.
2. Errors and response times return to their usual level within a few minutes.

7.4.1.4 If It Does Not Work

• If the previous version fails too, the cause is not the release: check the database, the secrets and the services this one calls.
• Call the other platform engineer. If customer data may be affected, follow Incidents.

7.4.1.5 Afterwards

• Open an issue for the fix. The fix reaches prod as a normal pull request, through stage (see Branching & Promotion).
• Write a postmortem if customers were affected.