Skip to content

3.4 Logging & Observability Standards

When something goes wrong in production, nobody can attach a debugger. What we know is only what our systems told us while they ran. Observability is that information, in three forms:

  • Logs: individual events, such as "sync run 812 committed 14 files".
  • Traces: the path of one request or job through every service it touched, with the time spent in each.
  • Metrics: numbers over time, such as requests per minute or the age of the oldest message in a queue.

This page fixes how every Moonitoring system produces them. How to report a failure, and how serious each kind of failure is, is in Error Handling.

3.4.1 OpenTelemetry Is the Standard

Decision

Every Moonitoring system produces logs, traces and metrics through OpenTelemetry (OTel). Code uses the OpenTelemetry APIs and conventions, never the SDK of a monitoring vendor directly. Errors and crashes also reach Sentry (decided 2026-11-04), but only through the shared library: code still writes nothing but tracing and logging events.

OpenTelemetry is the open standard for telemetry, supported by every major cloud and monitoring tool. Writing to it once means the backend can change, from Google Cloud to another tool or to both, without touching the code.

Rust Python
Logs and spans in code tracing logging and the OpenTelemetry API
Bridge to OpenTelemetry tracing-opentelemetry opentelemetry-sdk
Export opentelemetry-otlp opentelemetry-exporter-otlp

Every service sets up telemetry through one shared library per language, so that formats, levels and attributes are the same everywhere. A service does not configure exporters on its own.

3.4.2 Where Telemetry Goes

flowchart LR service[Cloud Run service or job] logging[Cloud Logging] trace[Cloud Trace] monitoring[Cloud Monitoring] sentry[Sentry, EU region] service -- "logs: JSON on stdout" --> logging service -- "traces: OTLP" --> trace service -- "metrics: OTLP" --> monitoring service -- "ERROR, FATAL, panics, crash dumps" --> sentry
  • Logs are written as JSON, one event per line, to standard output. Cloud Run sends them to Cloud Logging without any agent.
  • Traces and metrics are exported with the OTLP protocol to Google Cloud's Telemetry API (telemetry.googleapis.com), which feeds Cloud Trace and Cloud Monitoring. Services export directly; an OpenTelemetry Collector is added only when a need appears, such as sending to a second backend.
  • Errors and crashes are sent to Sentry by the shared library, from every service, worker and installed client, and grouped there into issues by release, as described in From Error to Alert. The same entries stay in Cloud Logging with the rest of the logs.

Application traces and metrics belong to the component's Platform or Farm project. Terraform declares their destination and grants the runtime identity the corresponding export capabilities. The Cloud SQL proxy has a separate metric stream to the database project, described below; granting export in Platform does not authorize that stream in the database project.

3.4.3 What Every Signal Carries

Every log entry, span and metric carries the same resource attributes, so that any signal can be traced back to the exact build that produced it:

Attribute Value Example
service.name The service, as named in its project farm-api
service.version The release tag, or the commit SHA before a tag exists (see Releases & Versioning) 1.4.0
service.instance.id Opaque identifier for this running process; stable until it stops, different for another process or restart A UUID v4 generated by the shared library
deployment.environment.name The environment stage
cloud.region The region europe-west8

The shared telemetry library constructs one Resource for traces and metrics, generating service.instance.id once with UUID v4 and reusing it across both providers. Do not generate a new ID per export or request. An explicit OTEL_RESOURCE_ATTRIBUTES=service.instance.id=... override is allowed only when the caller guarantees a nonempty, unique ID for that process. Do not put a fixed ID in Terraform or an image, and do not use K_REVISION: multiple instances share one Cloud Run revision. These rules follow OpenTelemetry's service-instance convention.

Google maps OTLP metrics to prometheus_target. It requires nonempty location and instance labels; cloud.region and service.instance.id supply those values in our contract. Missing instance identity can therefore reject metrics even while traces arrive. Google Telemetry API resource mapping

Other attributes follow the OpenTelemetry semantic conventions (http.request.method, db.system.name, exception.type and so on) instead of names of our own. Our own attributes use the moonitoring. prefix, such as moonitoring.sync_run.id.

3.4.4 Logs

  • Structured, never free text alone. The message is short and fixed ("sync run committed"); the variable parts are attributes (files=14, sync_run_id=812). Fixed messages can be counted and searched.
  • Every entry has a severity from the standard levels in Error Handling.
  • Every entry inside a request or job carries its trace and span IDs, so the logs of one request can be read together in Cloud Trace. The shared library adds them.
  • Never in a log: secrets, tokens, passwords, personal data (names, e-mail addresses, phone numbers, addresses) and raw payloads of farm files. Identify things by their IDs.
  • Logs are not storage. Cloud Logging keeps logs for 30 days. Every fact needed to explain a piece of data or a sync is kept in the database or in the bucket, never only in a log.

3.4.5 Traces

  • Every incoming request, message and scheduled job starts a trace, or continues the one it received.
  • The trace context travels with every call: in the W3C traceparent header over HTTP, and as a message attribute on Pub/Sub, so a sync can be followed from the Ingestion API to the parser worker.
  • Calls to the database, Cloud Storage and external APIs are spans of their own.
  • Sampling: every trace locally and in stage; in prod, 10% of traces, following the decision of the trace's first service. The rate is a configuration value, not code.

3.4.6 Metrics

  • Every endpoint and every job reports rate, errors and duration: how many, how many failed, how long they took.
  • Queues and pipelines report their backlog: messages waiting and the age of the oldest one.
  • Business events that matter for operations are counted too: sync runs committed, parse jobs failed, files skipped.
  • Metric attributes have few possible values, such as the environment, the feed type or the result. User, animal and request IDs never become metric attributes: they belong in logs and traces.

The process identity above is a resource attribute needed to distinguish concurrent writers. It is not permission to attach arbitrary request or business identifiers to metric points. Keep it stable for the process lifetime, and include instance churn in cardinality and cost checks.

Cloud SQL proxy metrics

Cloud SQL Auth Proxy v2.27.0 enables built-in client metrics by default. They use cloudsql.googleapis.com/client/connector/* with resource cloudsql.googleapis.com/InstanceClient, target the SQL instance's project, and authenticate as the proxy's identity. They are distinct from application OTLP metrics and from the proxy's optional older telemetry exporter. Missing metric permission does not prevent SQL connections. Pinned proxy documentation

Moonitoring makes this cross-project permission opt-in per component through system_catalog.<system>.sql_proxy_metrics_exporters, defaulting to an empty set. The selected identity receives only the custom database-project permission monitoring.timeSeries.create; it does not receive metric reads, Trace access or additional SQL privileges. The project comes from the configured SQL destination, not an application-supplied project ID. The generated database contract exposes proxy_metrics_enabled and proxy_metrics_project_id.

Access runtime and migrate are the applied initial identities. The extended catalog also permits a selected extra component only after its explicit sql_access profile and reviewed system baseline are enrolled; selecting proxy telemetry never creates a SQL login or grants table access. New component permissions require their own apply and private acceptance. When the application and sidecar share a Cloud Run service account, IAM grants apply to that identity, not only to one container.

The manifest must match the decision: set --enable-builtin-metrics=true only for an opted-in identity and --application-name=<component-name> to attribute the proxy's application_name label. Access uses access-backend and access-migrate respectively. If built-in metrics are not wanted or authorized, set --enable-builtin-metrics=false; selecting an application name does not grant permission. --disable-metrics controls the older exporter and is not a substitute for that flag. A catalog entry alone does not update application-owned manifests.

3.4.7 Telemetry Acceptance

A healthy service or a successful deployment does not prove metric ingestion. For a new component, or a shared exporter change:

  1. Test the common Resource: a generated nonempty ID is reused by traces and metrics, two processes have different IDs, and a valid explicit override is preserved. Check the actual packaged dependency, not only the library checkout.
  2. Review the destination and IAM delta: application telemetry stays in Platform/Farm; proxy metric permissions reach only the selected SQL identities in the database project. Preserve runtime/migrator separation.
  3. After an approved stage release, exercise the identified revision and record its image digest, service account, time interval and request/job result. Inspect a trace and an actual metric time-series point in the intended project; record the resource labels and timestamp. A metric descriptor alone is insufficient. Distinguish platform health probes from an external authenticated user request.
  4. If proxy metrics are enabled, verify fresh connector metric points in the database project separately. Allow an export interval with the process running and CPU available; a short-lived Job or a scaled-to-zero service may produce no periodic sample. Do not keep minimum instances running just to make a test pass without an explicit cost decision.
  5. Inspect export errors over the same interval. An OTLP InvalidArgument calls for resource/payload/schema inspection; a proxy PermissionDenied calls for its own destination/identity check. Neither authorizes a broad IAM grant. Save sanitized evidence and any remaining gaps; alarms and recovery need their own tests.

The 7 November baseline register records the two initial failures and final Access acceptance, including its limits. This standard describes the required behavior; each new component or release still needs its own evidence.

Not covered yet

Installed applications, such as the farm sync client, run outside Google Cloud. How they report their telemetry and their crashes will be added to this page when it is decided.