9.4 Secrets Management
A secret is any value that grants access by itself: a password, an API key of a provider, a token, a signing key. Whoever holds the value holds the access, so a secret must be kept where only the service that needs it can read it, and where every read is recorded.
The best secret is the one that does not exist. Wherever Google Cloud can prove an identity for us, we use that instead: services and pipelines get short-lived tokens, and service account keys cannot be created at all (see Identity & Access). Databases included: services sign in to Cloud SQL with their own identity, so there is no database password. What remains are mostly the keys of systems outside Google Cloud, such as the model providers and the vendor clouds.
The CI uploads client builds with no key either (decided 2026-11-13): its Google service account passes Identity-Aware Proxy on the Console's host with a short-lived Google ID token, and access exchanges that identity for the staff token of the staff service account linked to it (Service Accounts That Sign In Through IAP). No access-staff-<id> secret has to be written for it, and no staff token is kept in the CI.
9.4.1 Where Secrets Live¶
| Place | Holds | Who reads it |
|---|---|---|
| Secret Manager, in the project of the service that uses it | Every secret a running system needs, one secret per service and purpose | Only that service's account, through the secretAccessor role on that one secret |
Secret Manager in moonitoring-cicd |
The token of the connection to the code host | Cloud Build |
| The company password manager | The recovery copy of secrets that cannot be issued again (see below) | The secret's owner and, in an emergency, the organization administrators |
A secret's name starts with the system that owns it (decided 2026-11-11): <system>-<purpose>-<id>, such as access-sa-<id> for the credential of an Organization's service account, access-staff-<id> for a staff service account's and farm-feed-<id> for a vendor feed that farm pulls. IAM conditions then scope what a runtime may do to its own prefix, so one system's runtime cannot write the secrets of another system in the same project.
Secret Manager keeps every version of a secret and records every read in the audit log, which is copied to the locked archive.
A secret is never stored in code, in a committed configuration file, in the Terraform state, in a chat or in an e-mail. The CI runs on Cloud Build, so no secret is kept in GitHub either.
9.4.2 Owners¶
Every secret has an owner: the person responsible for the service or the integration that uses it. The owner:
- Creates the secret in Secret Manager and grants access to the service that needs it, and to nothing else.
- Keeps a recovery copy in the company password manager when the value cannot be issued again by its provider, because a migration or an incident may need it. Personal password managers and personal encrypted folders are not used: the company must be able to reach a secret when its owner cannot, and to revoke it when they leave.
- Rotates it when needed (see below), and hands the ownership over before leaving a role.
9.4.3 Rotation¶
Rotate a secret, that is replace it with a new value and revoke the old one:
- At once, when it may have leaked: it appeared in a log, a commit, a screenshot or a message. Revoke first, investigate afterwards.
- When a person who could read it leaves the company.
- At least once a year for keys of external providers that do not expire by themselves.
A service reads the latest version of its secret at start-up, so a rotation is a new version followed by a new deployment.