3.5 Error Handling

Every system fails sometimes. What matters is that a failure is noticed by the right person, at the right urgency, with enough context to fix it, and that it is never hidden. This page defines how serious each kind of event is, how an error is reported through the logging system, and when it becomes an alert. How logs are produced and where they go is in Logging & Observability Standards.

3.5.1 Severity Levels

Every log entry has one of six levels. The level says who must act, and how fast:

Level	Means	Example	In prod	Who acts
TRACE	Step-by-step detail of the code's execution	Each row read while parsing a file	Off	The developer, locally
DEBUG	Detail useful to diagnose a problem	The query sent for a dashboard tile	Off; switched on for one service, for a limited time	The developer investigating
INFO	A normal event worth recording	A sync run committed 14 files; a job started and finished	On	Nobody
WARN	Something unexpected happened, but it was handled and the operation went on	A retry after a timeout; a value skipped by the parser; a slow dependency	On	Nobody at once; reviewed when the rate grows
ERROR	An operation failed and was not completed	A parse job given up after its retries; a request answered with a server error	On	An engineer, the same working day
FATAL	The service cannot go on, or data is at risk	The service cannot start; an invariant of the data model is broken	On	The on-call engineer, at once

The same levels map onto every tool we use:

Level	Cloud Logging severity	OpenTelemetry severity number	Rust tracing	Python logging
TRACE	DEBUG	1–4	trace!	logging.DEBUG
DEBUG	DEBUG	5–8	debug!	logging.DEBUG
INFO	INFO	9–12	info!	logging.INFO
WARN	WARNING	13–16	warn!	logging.WARNING
ERROR	ERROR	17–20	error!	logging.ERROR
FATAL	CRITICAL	21–24	error!(fatal = true, ...)	logging.CRITICAL

Rust's tracing has no fatal level: an error! event with the field fatal = true is written as FATAL by the shared telemetry library.

3.5.2 Expected and Unexpected Errors

Not every error is a problem of ours. The difference decides the level:

• Expected errors belong to the normal life of the system: invalid input, a record that does not exist, a missing permission, a file the feed already delivered. They are returned to the caller as typed errors (Rust Result with an error enum, Python exceptions of our own) and logged at INFO or WARN at most. They are never ERROR: an ERROR must always mean something for us to fix.
• Unexpected errors mean something is broken: a bug, a dependency that does not answer after its retries, a rule of the data model violated. They are logged at ERROR, or FATAL when the service cannot go on.

3.5.3 Reporting an Error Through Logging

1. Add context while the error travels up, and log it once, where it is handled. Each layer adds what it knows (Rust: ? with .context(...); Python: raise ... from err) and does not log. The layer that decides what happens next logs the error once. An error logged at every layer appears five times and hides the real count.
2. Write the error as attributes, following the OpenTelemetry conventions: exception.type, exception.message, and for unexpected errors exception.stacktrace. Add the IDs that locate the problem, such as moonitoring.parse_job.id or moonitoring.sync_run.id. The trace ID is added by the shared library.
3. Retries are WARN, giving up is ERROR. Each failed attempt is a WARN; the ERROR comes once, when the operation is abandoned, for example when a parse job moves to parse.dead.
4. Never swallow an error. An error that is caught is either handled, logged at its level, or passed on with context. An empty catch or a discarded Result is not allowed.
5. Crashes are reported too. A panic hook (Rust) and sys.excepthook (Python), installed by the shared library, log any uncaught failure as FATAL and send it to Sentry before the process exits; Cloud Run then starts a new instance. An installed client, such as the farm sync client, also sends a native crash dump to Sentry, so a crash on a farm computer can be studied without reaching the farm.
6. No personal data in messages or attributes, in errors as in every log (see Logs).

RustPython

use anyhow::Context;

fn parse_file(job: &ParseJob) -> anyhow::Result<()> {
    let raw = storage.read(&job.data_path)
        .with_context(|| format!("reading raw file of job {}", job.id))?;
    parser.parse(&raw).context("parsing raw file")?;
    Ok(())
}

// The worker decides what happens next, so it is the one that logs.
if let Err(err) = parse_file(&job) {
    if job.attempt < MAX_ATTEMPTS {
        tracing::warn!(moonitoring.parse_job.id = %job.id, exception.message = %err, "parse attempt failed, retrying");
    } else {
        tracing::error!(moonitoring.parse_job.id = %job.id, exception.message = %err, "parse job abandoned");
    }
}

def parse_file(job: ParseJob) -> None:
    try:
        raw = storage.read(job.data_path)
    except StorageError as err:
        raise ParseError(f"reading raw file of job {job.id}") from err
    parser.parse(raw)

# The worker decides what happens next, so it is the one that logs.
try:
    parse_file(job)
except ParseError:
    if job.attempt < MAX_ATTEMPTS:
        logger.warning("parse attempt failed, retrying", extra={"moonitoring.parse_job.id": job.id})
    else:
        logger.exception("parse job abandoned", extra={"moonitoring.parse_job.id": job.id})

3.5.4 From Error to Alert

Sentry groups errors and crashes into issues (decided 2026-11-04): one issue per kind of error, with its count, the releases it appears in, the first and last time it was seen and the trace that led to it. The shared library sends every ERROR and FATAL event, every panic and, from installed clients, native crash dumps; it tags each event with the service, its version and the environment, so an issue shows which release introduced it and whether a fix removed it. Sentry is the only place that alerts on errors, so the same error never pages twice.

Level	What happens
WARN	Counted. An alert fires when its rate passes the threshold set for that service
ERROR	Grouped in Sentry. A new issue, or one that comes back after a release marked it fixed, notifies the platform engineers
FATAL	Alerts the on-call engineer at once

Sentry receives what explains an error, never the data it was working on:

• Its EU data region only, with personal data sending switched off and IP addresses not stored.
• No request or message bodies, no farm values, no tokens: the shared library scrubs them before an event leaves the process; identifiers such as moonitoring.parse_job.id are what lead back to the data, which stays in our systems.
• A failed integration is not a crash: a file or a source version we cannot read becomes an integration gap with its sample in our own bucket, never a Sentry attachment.

Every alert links to the runbook that says what to do.