4.11 Application Delivery

The common stage foundation lets Jacopo and Lorenzo develop different systems against one configuration and release procedure. A new component in a supported profile should need a catalog entry, scoped access and conforming manifests, without copying Access-specific Terraform or release scripts. An ordinary code release should reuse those resources.

Implementation boundary — 9 November 2026

Access exercised the common release path, including migration-before-backend and fresh telemetry. Farm subsequently proved onboarding and private cross-project invocation with commit fc5d06e; that evidence does not certify later Farm code. The 10 November runtime-configuration update prepares the new backend dependencies and a reusable pre-release guard. Runtime IAM, applied infrastructure and application rollout have separate evidence. Console, public edge and production remain separate integrations. See the current operations register and the Farm receipt and current limitations.

Release autonomy is not infrastructure self-enrollment. Jacopo and Lorenzo coordinate the first registration and any new resources or permissions. The service developer owns Dockerfiles, build/test scripts, manifests, hooks and migrations; trusted Cloud Build recipes and release tooling remain centralized. An accepted component can then release ordinary code changes without Terraform.

“Moonitoring Platform” names the ecosystem; moonitoring-platform-stage is one Google Cloud project hosting common runtimes. A repository contains source for systems/components, and a Terraform module is reusable provisioning code. These are different boundaries: neither one repo per project nor one module per service is required. See the system overview.

4.11.1 Objects and owners

A system owns a domain; a component is an executable Service or Job; a repository can contain several components or just a library. Projects host systems and are not created for every repository. A library is included in its consumers' images; it is not a central running service.

Owner	Owns	Does not own
Terraform / infra repository	Projects, network, SQL baseline resources, service accounts, scoped IAM, storage/queues, build connections/triggers, registries and delivery pipelines/targets	Application source, schema migrations, successive application revisions
Application repository	Code, tests, pinned dependencies, Dockerfiles, manifests, schema migrations and application permissions	Granting itself cloud permissions or changing another system's database roles
Cloud Build	Isolated checks, image publication and the trusted release preparation step	Approving its own rollout or applying arbitrary application-requested IAM
Cloud Deploy	Rendered application Service/Job delivery, approved predeploy and rollout	Terraform state, general project administration or implicit cross-system transactions
Workspace / Cloud Identity	Authorized changes to people in existing groups	Application roles, tenant membership or PostgreSQL permissions

Terraform and Cloud Deploy must not both own the same Service/Job specification. The legacy Terraform runtime module stays inactive for workloads owned by delivery. A new permission or resource requires a small reviewed infrastructure change; avoiding frequent copy/paste is the goal, not avoiding all future Terraform changes.

The system owns its data model and versions schema migrations with its code. A separate migration Job applies those migrations; backend and worker identities only use the permitted data. Runtime identities receive no schema ownership or DDL privileges and do not migrate on application startup. Separation of execution and privilege does not require a separate repository or manual database edits. Each system has one coherent migration history for its schema; sharing the Cloud SQL instance does not make Access and Farm share a schema or migrator.

Decided on 2026-11-11, for every component that touches a database:

• One SQL role per component. The backend, the consumer, the relay, the sweeper, the bootstrap-admin job and the Dev ORM each have their own role with only the grants their work needs: <system>_runtime is the backend's own role, never shared with another component; only the migrator holds the owner role. The names are in 4.11.2, and the baseline carries the complete grants of every role.
• A runtime that writes to the database checks the schema version at startup, with one shared helper in moonitoring-common-rs (SchemaRequirement of moonitoring-db, decided 2026-11-11): each component names what it needs (the tables and columns it reads and writes, the privileges its own SQL role must hold, the sequences it uses) and the helper checks them in the database catalog before the component serves or takes work. An unmet need or an unreadable catalog stops the process, with every unmet need in the log. A later schema that only adds is accepted, so a revision of the previous release may cold-start after the next release's migrate job has run; dropping or renaming something a component uses blocks it. The check reads only the catalog: runtime roles have no privilege on seaql_migrations. No runtime runs DDL: creating or changing tables, indexes, roles or grants happens only in the system's migrator.

4.11.2 Registration contract

The executable configuration belongs to the infra repository. The application receives the generated contract and references; it does not invent cloud IDs, audiences or service-account names.

Define the common fields and supported profiles now, then fill in each component while developing it and before its first deployment. Complete Farm, Releases and Console implementation sheets are not prerequisites for parallel development. A short descriptive component sheet records the system/component and owner, source location, build/delivery profile, catalog references, required data and capabilities, callers/destinations, schema compatibility, operational limits and acceptance evidence. It is not another executable configuration format. The catalogs and validators remain authoritative.

Declaration	Required meaning
System/component and repository	Stable identifiers, source repository and build profile; components that share a repository remain distinct runtimes
Destination	Stage project, region and Service/Job identity; preserve existing IDs
Resources and capabilities	Database/schema, secret references, bucket, queue, key and destination services; resource creation and permission grant are separate
Delivery	Pipeline/target, workload, image aliases, manifest/file allowlist, runtime and migration identities, approver scope
Capacity and operations	Instance/concurrency/connection limits, Job timeout/retry, telemetry and recovery responsibilities

One delivery per component (decided 2026-11-11): the backend, the consumer, the relay, the sweeper and each job of a system are separate deliveries, each with its own release, even when they share a repository and a version tag; the migration Job stays in the delivery of the backend, which runs it first. Rules for the identifiers, and the names decided on 2026-11-11:

• A delivery ID is short, at most 11 characters, such as acc-relay or rel-sweep, because it becomes part of cloud resource names.
• Each delivery has one runtime requirements file, deploy/runtime-requirements/<delivery-id>.json, covering exactly the components of that delivery. One file for the whole repository is refused.
• One SQL role per component that writes, named <system>_<component>: the backend keeps <system>_runtime, the migrator <system>_owner, and the others are access_relay, access_consumer, access_sweeper, access_bootstrap_admin, farm_relay, farm_consumer, farm_sweeper, releases_relay, releases_consumer, releases_sweeper and so on for every system. Roles and their IAM logins are created by infra in the SQL baseline (v3), each logging in with its component's service account, never owner and never DDL; each system's stage_grants.sql gives every role only the tables it really uses (e.g. a relay only the outbox), in the same pull request as the migration that needs it, and none on seaql_migrations.
• In the repository, each delivery has deploy/skaffold/<id>.yaml with only that component's manifests, next to deploy/runtime-requirements/<id>.json; access and farm may keep their current files. Cloud Run Service and Job names and image names stay the components' own (access-relay, farm-consumer); the short ID names only the pipeline, its target, bucket and CI/CD identities, so renaming it later means recreating them.
• An image is built from the folder of its part, with underscores turned into hyphens: repo/access-bootstrap_admin builds the image access-bootstrap-admin.

System or app	Delivery IDs (agreed with infra, 2026-11-11)
access	access (backend and migrator), acc-relay, acc-cons, acc-sweep, acc-boot (the job access-bootstrap-admin)
farm	farm (backend and migrator), farm-relay, farm-cons, farm-sweep, later farm-parse
releases	releases (backend and migrator), rel-relay, rel-cons, rel-sweep
status	status (backend and migrator)
news	news (backend and migrator), news-relay, news-cons, news-sweep
Apps	console (Console server), account (Account app), dashboard (Organization Dashboard server)

A system's Dev ORM is not a stage delivery for now.

The existing service_contract describes runtime resources and granted capabilities. The P1 application_delivery_contracts output supplies version 1 of the trusted release contract used by the common packer (--contract <file.json>). These are distinct contracts: one describes runtime integration, the other validates release provenance and its destination. Neither proves that application permissions or cloud access have passed their tests. Do not put secret values, tokens or Terraform state into an application manifest or shared contract file.

Stage also exports application_runtime_contracts. The common check_onboarding.py compares a selected bootstrap delivery entry with its stage runtime entry: destination, deployer, workload set, system, component service accounts and SQL profiles must agree. Use --preparation-only only before enabling the stage release grants; the final pre-release check requires those grants to be declared. The output explicitly reports configuration matching, not verified live permissions. Bootstrap gate/publisher identities precede stage bindings; bootstrap release activation precedes runtime_deliveries.release_enabled=true. Registry publisher grants are declared separately for Platform or Farm.

Telemetry is part of the runtime contract: application metrics and traces go to Platform/Farm, while optional built-in SQL proxy metrics go to the database project through sql_proxy_metrics_exporters. The common library supplies a process-unique Resource; manifests must not substitute a revision-wide instance ID. Apply the telemetry contract and acceptance checks when registering a component or updating the shared library. A generic delivery profile does not bypass SQL-baseline validation or prove metric ingestion.

P1 candidate records include contract_sha256. Older records must be repacked rather than overwritten; previously verified publication images may be reused if their commit and manifests are still valid for the selected release. The new rendered-manifest guard intentionally adds read access for the release identity to the delivery/ prefix of its own artifact bucket. That IAM delta must be reviewed in the real plan; resource preservation does not mean that every IAM binding is unchanged.

Supported CI code includes dependency, rust-library, rust-system and the rust-slint-app profile for Console's server/WASM build with pinned Design, Common and Docs dependencies; a nuxt-app profile for the Account app is proposed in infra and not yet merged. Adding a profile does not prove a successful cloud build or deployment. Delivery supports service-with-migration (backend and its migrator), SQL-free service/job, service-read-only for a Service with a prepared read_only SQL login, and, since 2026-11-11, service-runtime-sql and job-runtime-sql for a relay, consumer, sweeper or operational job that writes with its own SQL role and no migrator. The pipeline releases a job-runtime-sql Job but never runs it: only infra-admins execute it, and every execution raises an alert. These profiles do not accept arbitrary languages or layouts. Events and Scheduler require separate resource/invocation configuration, not just a generic Job. Native apps, static sites and permanent workers need explicitly supported paths.

SQL registration requires both enrollment and a reviewed baseline receipt. Extra components default to no SQL; component (the component's own <system>_<component> role) and read_only are explicit choices, and only the separate migrator can receive the owner role. A component role needs a v3 baseline whose receipt lists it in component_roles. Read-only selects reviewed tables and does not enforce tenant filtering. The v3 procedure can extend migrated Access without rewriting its v2 checksum or ownership, but its local PostgreSQL tests do not replace private Cloud SQL acceptance.

Additional writable components use the runtime-SQL profiles (infra, 2026-11-11): one delivery per component, its own SQL role, no migrator and no DDL. The system's migrator, released with the backend, stays the only one; each component checks the schema it needs at startup (4.11.1) and is released after the migration it depends on. No general orchestrator or simultaneous deployment is required. The profiles are implemented but have not had their first live acceptance: the first relay, consumer and sweeper releases after the stage reset are that acceptance.

4.11.3 First registration and later releases

Runtime configuration is a versioned contract

The application declares what it needs; the stage manifest is the approved configuration for running it. Terraform owns the referenced resources and IAM. Ordinary development follows this configuration. A new dependency requires a coordinated change to requirements, manifest and scoped infrastructure access, not a silent fallback to a local emulator or an administrative role.

The opt-in runtime_requirements delivery field references the allowlisted deploy/runtime-requirements/<delivery-id>.json of that delivery and approved non-secret environment bindings per image component. Bootstrap and stage export matching bindings. The application JSON declares the target container, required environment values and their types/allowed values, and forbidden local overrides. It travels in the immutable source bundle and its hash/provenance checks. It does not grant IAM.

For enrolled components the common tooling checks source manifests before publication/packaging and the rendered manifests before the candidate becomes usable. Removing a required declaration cannot remove a trusted binding. Existing deliveries without this field retain their current contract and checks; they must opt in before claiming this additional runtime-configuration coverage.

The Rust publisher also runs the built backend with the fixed --check-config mode in an isolated container without network, cloud credentials or host mounts. This mode uses the same pure configuration validation as normal startup and returns before telemetry, database connection and other side effects. The publisher verifies that the published image has the tested image configuration. No application command is executed by the privileged release verifier. The application test suite must also exercise its actual binary with the manifest values and negative cases; a JSON declaration alone cannot discover a requirement forgotten in the code's validator.

These checks detect missing or inconsistent configuration, not live access. Private routing, IAM, PostgreSQL privileges and application authorization still need their own acceptance. Secret values must not enter these files or build logs; use approved secret references and the appropriate runtime profile.

Farm is the first opt-in: token issuer, private Access URL/JWKS, Google identity, raw/dataset bucket names and Secret Manager project are explicit. The dataset name is required by configuration even though the current backend does not use dataset objects, so no dataset IAM is granted merely for declaring the name. See the implementation and permissions.

Register a new component once:

1. Select a supported profile and declare the repository, destination, identities, resources and exact capabilities. Decide which operations the application permits independently of IAM.
2. Review the Terraform plan. It must not recreate Access or unrelated resources, broaden existing permissions or activate production. Apply it only after the normal infrastructure review; retain generated contract outputs.
3. Add the conforming Dockerfile, manifests and tests to the application repository. For SQL, prepare the scoped baseline before migration and verify runtime privileges on the created objects afterward.
4. Check render, artifact-to-workload-to-identity mapping, permissions and first-creation handling. Temporary project-level creation grants must expire and be replaced by named-resource grants once the workload exists.
5. Complete the first approved release and record the result, including denied cross-system access. A fixture proves its profile, not the functionality of a future application.

Release an existing component:

Reviewed commit and pinned dependencies
  → isolated checks and approved publication
  → immutable images and publisher evidence
  → candidate bundle from the exact Git commit
  → trusted release preparation and render
  → stage rollout approval
  → missing migrations, when configured
  → application revision and required checks

The release trigger remains manually initiated, source-free and approval-gated, using trusted infra code rather than arbitrary repository build instructions. The stage_release.py operator command composes publisher validation, immutable packaging/upload and release preparation from the applied contracts. Its init/advance commands never approve a build or rollout, create IAM, or apply Terraform. stage_release.py status reads cloud state; advance may upload a candidate and start a build. An ambiguous API reply stops for reconciliation instead of starting another build. The separate stage_approval.py inspect/approve command submits an explicit human approval for Cloud Deploy after inventory/provenance checks, using the approver's own account; Cloud Build approvals remain separate. That new helper is tested and packaged offline; its first live use remains distinct from the accepted Access/Farm release runs. See the operator runbook.

The CLI currently requires the reviewed SHA to match current remote main, both before starting publication and before candidate upload. A previously merged SHA is not automatically an admissible new candidate. The operator kit freezes the selected applied delivery, runtime and operator outputs, compatible tooling and provenance; it never distributes Terraform state or credentials. Keep each run's input copy and ledger unchanged when issuing a newer kit. See Stage Release and Recovery.

No push/merge event is introduced implicitly and the publisher cannot deploy. Any later automatic orchestration must retain the same provenance, approval and failure checks.

A normal code-only release needs no Terraform apply. Changes to resources, privileges, exposure or supported execution profiles go through infrastructure review. Approving the Cloud Build release-preparation build and approving the Cloud Deploy rollout are separate decisions; the latter authorizes migration and deployment. Follow the common onboarding guide, using the Access runbook for its specific migration and first-creation checks.

4.11.4 Boundaries the procedure must preserve

• Provenance: candidate files come from the exact published commit; images use verified digests. Preserve checks for Git objects, file allowlists, publisher identity, approval and completion, registry evidence and the delivery target. A tag or a trusted repository name alone is insufficient.
• Identity: checker, publisher, release creator, renderer, deployer, runtime and migrator keep distinct duties. A valid migrator identity on a backend manifest must still be rejected. Rendering has no reason to mutate the database.
• Schema compatibility: the available schema constrains which versions can run. Add compatible structures before the code that needs them; remove old structures only after active versions and the recovery path no longer need them. A compatible code-only patch need not change the schema or redeploy other components.
• Migration failure: apply only missing migrations with safe locking/retry. Failure prevents the dependent new revision; it does not automatically stop the old backend. A migration lock does not serialize normal application writes or the whole release. Transactions and idempotency remain application responsibilities.
• Resumption and retry: resume the same immutable run without duplicating builds. A failed Cloud Deploy job uses a distinct recovery procedure, after its previous Cloud Run Operation/Execution has been reconciled. A caller timeout does not prove that work has stopped. Never ignore a failed migration.
• Release ordering: version 1 permits one executing rollout per component/environment, including retries and recovery. Operators may prepare multiple candidates but must agree on the current one and reject superseded pending candidates. The explicit approval helper checks the same pipeline/target, recent unchanged snapshot and absence of active, unknown or newer undiscarded rollouts before approving. “Newer” refers to Cloud Deploy rollouts, not Git commits or unpublished bundles. This is still not an atomic check-and-approve or a distributed lock; IAM also permits approval through the console.
• Application recovery: use a reviewed fix/revert on current main, preserving applied migrations, through the normal pipeline. Version 1 does not use Cloud Deploy rollback to an earlier release. SeaORM already rejects an applied migration unknown to the migrator; regression tests make this boundary explicit. Keep applied migration files and history in the new main commit. No release automatically undoes SQL or restores the database.
• One-time stage reset before the single baseline (dated exception, decided 2026-11-11). The rules above on applied migrations hold from the first real stage release onward. Before it, and only once, each system replaces its migration history with one baseline (Releases and Versioning) and its stage database is emptied so the baseline applies to an empty schema. The reset is an infrastructure operation, not application code:
  • No destructive code in a migrator or a runtime. The system's repository only provides the baseline migration and its stage_grants.sql.
  • Infra runs it with its own runner (tools/sql-stage-check in moonitoring-infra) under the system's migrator identity, following a runbook approved by infra-admins: an on-demand backup first, then a read-only inventory whose hash the operator approves, then the drop of the objects owned by <system>_owner in public only, with RESTRICT. Never DROP OWNED BY, never CASCADE, never dropping or recreating the database.
  • Then the v3 SQL baseline creates the components' roles (4.11.2) and records them in its receipt, and the first approved release runs the system's migrator on the empty schema.
  • Stage only, synthetic data only, one time per system. Production never uses this path. Until the runbook and runner are merged in moonitoring-infra and a system has been reset, its stage database keeps the current history.
• Multiple components: a native Cloud Deploy delivery handles one Cloud Run workload per target/render. Backend, consumer and relay need explicit order and coordinated schema prerequisites, not an assumed atomic rollout. After a successful migration, a failing consumer is recovered independently. Google target limitations
• Service authorization: IAM proves permitted invocation at the cloud boundary. The destination enforces RPC, data and user-context policy; a subnet or service-account allowlist is not every application permission. A gRPC service's list of callers is authentication only (decided 2026-11-11): every RPC checks the calling identity for that operation and returns only the data that caller needs, as each contract states per RPC (gRPC Between Services).

4.11.5 Milestone and handoff

The initial common base closes four steps, with evidence for each:

1. Standard: component fields, examples, catalogs and validators agree; available profiles and extensions are explicit.
2. Operator procedure: a versioned kit, Access publication/release, approvals, interruption/retry and ordering of two candidates are usable by both intended operators. Direct personal Owner access is not proof of group-only access.
3. Generic onboarding and access: the real Farm backend/migrator and a temporary Platform probe have proved independent onboarding, SQL separation and cross-project network/IAM. Reuse those receipts; the complete Farm product or Console are not prerequisites.
4. Minimum operations: received alerts, resource/connection limits, a representative isolated restore and a tested compatible application recovery procedure.

Read-only delivery, events, Scheduler, generic Jobs and additional SQL writers have separate acceptance before their first ordinary use. Controlled fixture deployments are allowed to establish that acceptance. Console/staff web, product-client access, complete product systems and production are separate integrations; they do not block use of profiles already accepted. Any newly exposed path still needs its own session and authorization tests. The operations register tracks what has actually passed.

Growing from an initial group of roughly five farms to at least one hundred is the capacity planning baseline. It does not force particular providers, activate real-data onboarding during P0/P1 or make fixture tests evidence of capacity at that scale.