7.4.4 Stage Release and Recovery

Use this procedure for a registered component in an accepted common delivery profile. Access and Farm are the accepted reference integrations. First onboarding and new permissions are coordinated by Jacopo and Lorenzo; autonomy covers ordinary releases after that handoff. The procedure is independent of Console and uses Google Cloud's build, rollout and runtime views.

• When to use: publish a reviewed stage version, resume an interrupted operator run, or recover a failed release.
• Impact: the named stage component and its explicitly declared dependencies. This runbook does not authorize production, new IAM or schema rollback.
• Access needed: the configured infrastructure-operator permissions for the selected publication/release identities; stage approvers authorize Cloud Build and Cloud Deploy separately. Recovery needs its own scoped permissions, not merely approval membership.
• Acceptance: use the foundation register for dated evidence. The ordinary Access release and the isolated SQL-free fixture's failed-job retry with unchanged approved artifacts have succeeded, including private probes and cleanup. A restarted command found the same publisher without duplication, and unsafe/duplicate retry attempts were rejected. The explicit approval helper adds preventive ordering checks; its tests do not claim distributed locking or certify every SQL recovery case. Each operator checks their own effective actions during normal first use; a dedicated Lorenzo session is not a prerequisite and has not been claimed as tested.

7.4.4.1 Check the handoff

The infrastructure owner produces a versioned kit from the selected applied delivery, runtime and operator outputs and the matching infra tooling. It records provenance and hashes, and includes the instructions and dependency versions needed on another computer. It contains no Terraform state, credentials or application data. Follow the kit instructions; keep the supplied source commit and receipt with the kit.

1. Obtain the kit and its expected manifest hash through the agreed company-controlled handoff. Run operator_kit.py verify --kit <directory> --manifest-sha256 <expected-hash> from a trusted infra checkout before executing the kit's tools. Verification checks file hashes and cross-contract agreement; hashes identify contents but do not by themselves authenticate the sender. Never alter an exported JSON to pass validation.
2. Confirm project, region, component, runtime/migration identities, SQL profile and intended endpoint. The final onboarding check requires enabled release grants; --preparation-only is not an operational acceptance check.
3. Use your own authorized Google identity. Install the declared Python dependencies, Git and Google Cloud SDK, and use a separate application checkout. Keep the virtual environment and run directory outside the immutable kit; run packaged tools with python -B so they do not create bytecode files inside it. No operator needs full Terraform state.
4. Select the reviewed SHA at current remote main. The CLI checks it before publication and before candidate upload. If main has advanced, prepare an admissible candidate; do not bypass the check or substitute another SHA in a frozen run.
5. Choose a new run directory. Inspect other releases for the same component/target and agree which candidate is current. Independent components can proceed separately.

Each run retains its input copies and ledger. A newer kit is for new runs; it does not rewrite the evidence of an older run. After a first workload exists, including a non-Ready one, the infrastructure owner restricts the temporary creation grant to named resources and exports updated contracts for later runs.

7.4.4.2 Release

Use the kit's matching operator command, with your account and local application path:

Step	Expected outcome
init with the selected contracts, SHA and publication request	A publication build ID and its Cloud Build approval link
Approve publication, then advance	Successful verified publication, immutable candidate and a separate release-preparation build awaiting approval
Approve preparation, then advance / status	Rendered candidate and the Cloud Deploy rollout awaiting its approval
stage_approval.py inspect, then explicit approve with the approver's account	Approved predeploy, followed by deployment only if it succeeds
status and runtime checks	Successful rollout, expected digest/identity and Service Ready; required request and telemetry checks pass

stage_release.py init/advance/status does not approve gates. Cloud Build approvals remain separate. The approver explicitly uses the matching kit's Cloud Deploy approval helper:

python -B tools/stage_approval.py inspect --run-dir <run> --account <approver@moonitoring.com> --output <snapshot.json>
python -B tools/stage_approval.py approve --run-dir <run> --account <approver@moonitoring.com> --snapshot <snapshot.json> --coordinated-with-team

It checks the exact candidate/render, PENDING_APPROVAL/NEEDS_APPROVAL, the same pipeline and target, a snapshot no older than ten minutes, and no active/unknown or newer undiscarded rollout. It rereads state and records intent before POST. A newer rollout is a Cloud Deploy object, not a Git SHA or merely stored bundle. The snapshot records the approver separately from the original operator; the publisher account in the ledger is not rewritten. Existing IAM determines who can approve; no role is granted by the tool. Use the same approver account for inspect and approve. The helper rejects SDK impersonation, token-file and credential overrides rather than silently approving as a different identity.

This is an explicit manual approval, not a new automatic gate or an atomic check-and-approve. Console/API approvals remain possible under IAM, and separate computers are not serialized by a local lock. Coordinate the selected candidate; a lost response is reconciled before another request. An approval or a created execution is not a successful deployment.

For service-with-migration, a separate Job applies only missing migrations. Its migration identity can change the system's schema; the backend identity cannot. A migration error blocks the dependent new revision. The previously deployed backend is not automatically stopped. Keep schema changes compatible with active versions and the recovery path.

The version 1 ordering rule is one executing rollout per component/environment, including retry and recovery. Multiple candidates may wait, but operators record and reject those superseded by the chosen candidate. Reconcile any execution with uncertain status before approving another. The local ledger lock and migration lock do not enforce ordering between two operators on different computers.

7.4.4.3 Resume or retry

First inspect the ledger, the selected build/release/rollout and, where migration ran, its Cloud Run Operation/Execution. A timeout on the caller does not prove that the Job stopped. The recovery helper inventories the target and checks the declared Job before a retry; it does not grant permission, approve a rollout or provide a distributed lock.

Ordinary recovery assumes the declared workloads already exist with their scoped grants. A first deployment that failed before creating a workload needs the separate administrative first-creation procedure; 404 or 403 is not evidence that a retry is safe. Reconcile resources, executions and permissions before proceeding.

python -B tools/stage_recovery.py inspect --run-dir <run> --output <new-snapshot.json>
python -B tools/stage_recovery.py retry --run-dir <run> --snapshot <snapshot.json> --reconciliation <evidence.json> --phase-id <Cloud-Deploy-phase> --job-id <Cloud-Deploy-job> --coordinated-with-team

These paths refer to tools in a verified kit. Use the infra procedure for the reconciliation-file schema and checkout paths. Keep snapshots and receipts outside the immutable kit. Inspection is read-only in Google Cloud; retry rereads the target and requires a recent matching snapshot, the exact failed job of an approved failed rollout and no conflicting active or newer candidate.

Recovery is assisted. Even an empty list of active Executions does not exclude an earlier :run request with an uncertain result. The reconciliation file must identify the previous Operation/Execution result, or concrete evidence that the work never started, with verifier, time and reference. It is an attributed operational check, not an automatic certification or signature. If the result cannot be reconstructed, an authorized administrator must reconcile it; the new grants do not include project-wide Operations access. Elapsed time alone is not evidence that retry is safe.

Situation	Action
Local command stopped; build or rollout still running	Keep the run and use status; wait for a terminal result. Do not start a second build or execution
Build-start response was lost	Reconcile trigger, time and SHA/candidate with Google. Use adopt only for the matching build ID; the CLI verifies it before recording it
Same rollout has a failed, retryable Cloud Deploy job	Inspect the failure and reconcile the previous Operation/Execution or proven pre-start failure. Supply the reconciliation evidence to the checked retry procedure, with the same approved artifacts and exact rollout/phase/job IDs
Code, manifest or contract must change	Prepare a new reviewed candidate/run and retain the failed run as evidence
Candidate was superseded or rejected	Do not approve or resume it. Record its disposition and use the selected candidate

advance is not a failed-rollout retry. The retry operation uses Google's retry-job mechanism; a Cloud Deploy job ID is not the name of the Cloud Run migration Job. Never ignore a failed migration or repeat an uncertain execution blindly. A successful retry still needs the normal runtime verification. See Google's retry command.

The helper records intent before sending the retry. If the response is lost, do not resend it: create a new inspection snapshot and reconcile the JobRun/Execution with the receipt. A further failure has its own JobRun and needs a new inspection and cause analysis. Read failures never mean that no execution exists.

7.4.4.4 Restore previous behavior

The initial ordinary path is a reviewed fix/revert of application code on current main, preserving already-applied migrations, through the normal pipeline. Confirm that the resulting code remains compatible with the current schema; a complete repository rollback may remove migration history and is not equivalent. This path rebuilds and retains the normal approvals.

Version 1 does not use Cloud Deploy rollback to an earlier release or accept an arbitrary historical SHA. SeaORM already stops when the database ledger contains an applied migration unknown to the migrator; explicit regression tests cover that failure. Preserve the applied migration history when reverting application behavior.

The main-only fix/revert route does not reverse schema changes or restore the shared SQL instance. If data is damaged, stop improvising release changes and follow the separately authorized database-recovery procedure. A successful retry of the same release is not a test of restoration to previous behavior.

7.4.4.5 Verify and record

Keep a sanitized receipt with kit/contract hashes, infra and app commits, operator identity, approved candidate/digests, build/release/rollout and Execution IDs, times and results. Check the affected Service's readiness and the required authenticated operations, traces and metrics; reconcile any temporary IAM or fixture resources.

The handoff reuses the accepted operator flow and interrupted-run recovery. Automated cases exercise the approval guard against conflicting or superseded candidates; use an inspection before the next ordinary approval without injecting another cloud failure. Test failed migrations and recovery on isolated fixtures; do not introduce a broken migration into shared Access. Each operator uses their own authorized identity and records the actions exercised on normal first use. Shared intended permissions and service visibility do not prove every effective action; neither a duplicate release nor a dedicated Lorenzo session is required to continue. Do not treat personal Owner privileges as proof of group-only access. Reuse valid profile evidence and add the new component's own permissions/configuration checks.

If the result is uncertain, involve the other infrastructure owner with the receipt before another mutation. Update the foundation register only with what the evidence proves. Cross-project network acceptance, an isolated database restore, received alerts and additional execution profiles retain their own acceptance checks.

7.4.4.6 Isolated SQL-free recovery fixture

Status: accepted for the isolated SQL-free permission-failure case on 9 November; probes and cleanup completed. The test used temporary Service access-recovery, sourced from a separate standard-library-only fixture package in the already-connected Access repository. Its publisher and delivery pipeline are named recovery; its runtime is sa-access-recovery. Build/release identities, role bindings and candidate storage are dedicated to the fixture. It has no database or migrator and adds no fault hook or special precheck to the common delivery engine. The foundation register records the actual failed JobRun, restored role, unchanged artifacts, successful new JobRun and duplicate-retry rejection. The following procedure describes that bounded test; recreating the retired Service requires its own reviewed first-creation setup.

Step	Action and required evidence
Ready baseline	Apply the reviewed fixture resources, use normal release approvals and record a Ready baseline with expected image and identity. Confirm that the fault role is scoped only to this fixture; shared Access continues unchanged.
Controlled failure	With the current etag, remove only run.services.update from custom role stageRecoveryBackendUpdate. Release the next approved fixture candidate through the ordinary pipeline. Record a terminal 403 deployment failure and the unchanged Ready baseline. If the expected failure is not observed, stop rather than removing further permissions.
Restore and reconcile	Restore the role using a verified Terraform plan. Inspect the exact failed rollout/job and record concrete terminal-failure evidence; reconcile any uncertain operation before retry. Do not infer safety from an empty list or elapsed time.
Actual retry	Run the verified kit's stage_recovery.py inspect/retry with the reconciliation file and team coordination. Retry the same failed job and approved candidate; retain the retry receipt and new JobRun ID. No new candidate, manual deployment or ignored job substitutes for this test.
Verify and clean up	Record successful retry and expected revision Ready/image/identity; confirm ordinary Access is unaffected. Restore the role, remove the fixture Service, reconcile exists = false and nominal grants with Terraform, and close any first-creation IAM window. Preserve sanitized receipts and the final plan.

Keep the common modules' pipeline/target/bucket deletion protections enabled. Cleanup retains the inactive protected delivery definitions and audit artifacts, with automatic triggers disabled; it is not deletion of every GCP resource or a zero-cost guarantee. No new Cloud SQL instance is created for this test.

This closes only the tested deployment-permission recovery case. It does not test a failed SQL migration, schema rollback, production, candidate concurrency or the later caller-in-Platform → Service-in-Farm network path. Those retain separate acceptance criteria.