Skip to content

6.8.2.1 SyncRun

6.8.2.1.1 Purpose

One synchronisation of a feed, and what it covered. Comparing what the source sent with what we already hold reveals added, changed and removed records, including deletions made at the source. Its status says only whether the client delivered; whether we processed what it delivered is read from its ParseJobs. A run that fails before reaching our API is reported by the client at its next contact, so a feed with no run at all is silent, not failed.

6.8.2.1.2 Questions it answers

  • When was a feed last synchronised, and what did the run cover?
  • How many records were added, changed or removed?
  • Can this run prove that a record was deleted at the source?
  • What triggered the run?
  • Did the client deliver, and where did it stop if not?
  • Did we process everything it delivered?

6.8.2.1.3 Fields

Field Type Cardinality Description
id string (uuid) 1..1 Key.
source_feed SourceFeed 1..1
kind enum 1..1 Values: full_snapshot, incremental.
full_snapshot: everything the source has for the covered period. Only a full snapshot can reveal deletions, unless the source sends them explicitly (e.g. ICAR ADE isDeleted).
Our farm client sends incremental runs with explicit deletions, detected against its local cache; setup and a digest mismatch (SyncDigest) trigger a full_snapshot of the affected period.
coverage Period 1..1 The time span of the data the sync covers.
started_at datetime 1..1
finished_at datetime 0..1
status enum 1..1 Values: running, done, failed.
Delivery only: whether the client read the source and committed the raw files to us. Processing by the parsers is read from the run's ParseJobs (ParseJob), never stored here: pending while any job is pending or running; done when every job is done; partially_failed when some failed and some are done; failed when every job failed. The Console shows delivery and processing side by side.
added integer 1..1
changed integer 1..1
removed integer 1..1
unchanged integer 1..1
resource_scope CodedValue 1..*
filter_definition CodedValue 1..1 Versioned query/table/resource filters, not just a time period.
raw_inputs RawMetadata 1..*
source_order_key string 0..1 Source snapshot/change sequence; comparison semantics are adapter-defined, not arrival timestamp.
snapshot_verified boolean 1..1 Default: false.
True only after all pages/resources succeeded and source completeness/deletion semantics were established.
received_keys_manifest string 1..1 Immutable inventory of source keys before canonical filtering, so parse failures cannot simulate deletions.
trigger enum 1..1 Values: setup, schedule, digest_mismatch, manual.
setup: first sync after installation; schedule: SyncSchedule; digest_mismatch: SyncDigest differs.
sync_client SyncClient 0..1 The client that ran it; empty for feeds that reach us without our client.
attempt integer 1..1 Default: 1.
A retry of the same planned run is a new SyncRun with the same coverage and attempt + 1.
failed_step enum 0..1 Values: connect, fetch, verify_snapshot, store_raw, commit.
Where a failed run stopped, on the client's side only; only when status = failed. The values parse, diff and resolve were removed: they belong to ParseJob and to the processing state above. connect: the source or our API could not be reached; fetch: reading the source; verify_snapshot: completeness check; store_raw: upload to the bucket; commit: the commit call to our API.
last_error string 0..1 Why it failed, in one line a person can act on (e.g. "login refused by DelPro", "page 7 of 12 timed out").
failure_class enum 0..1 Values: source_unreachable, credentials_rejected, source_changed, unsupported_source, timeout, api_unreachable, storage_error, abandoned.
Set only when status = failed. Transient classes (source_unreachable, timeout, api_unreachable, storage_error) are retried by the client within the night (attempt + 1); credentials_rejected and source_changed are not retried and raise an issue for the Organization and for support.
unsupported_source: no client adapter (SourceAdapter) covers what the client detected; the run records an IntegrationGapOccurrence and is not retried until a client update brings an adapter.
abandoned: set by the farm sweeper, never by the client, when the run stopped reporting (see the note below).
deadline_at datetime 1..1 When the run must be finished, set at start from the feed's expected duration with a margin.
last_heartbeat_at datetime 0..1 Renewed by the client every few minutes while the run is running.
adapter SourceAdapter 0..1 The client adapter that read the source; empty when none covered the detected version.
detection SourceDetection 1..1 What the client found at the source before choosing an adapter (see SourceDetection).
detail_path string 1..1 Run log in the bucket next to the raw files (JSONL: step, resource, page, counts, timing, error). Stack traces stay in Cloud Logging; the facts that explain the run do not.
created_at datetime 1..1

6.8.2.1.4 Relationships

References

Referenced by

Uses types

Period, CodedValue, SourceDetection

6.8.2.1.5 Rules & constraints

  • One synchronisation of a feed, and what it covers. Needed to see changes and deletions made at the source.
  • Deferred report. A run that fails before the client can reach our API (connect, fetch, verify_snapshot, or store_raw with no commit) is recorded by the client locally and created at the next contact as a SyncRun with status failed, its failed_step, its failure_class and started_at as of the attempt. A feed whose schedule (SyncSchedule) passed with no SyncRun at all is therefore "silent", which the Console distinguishes from "tried and failed".
  • Sweeper: farm-sweeper, a Cloud Run service that Cloud Scheduler calls every five minutes (a service rather than a job since 2026-11-05, so nothing runs between calls), fails every running SyncRun whose deadline_at has passed or whose last heartbeat is older than ten minutes, with failure_class abandoned and a last_error a person can act on ("no heartbeat since 02:14"). A late commit of an abandoned run is refused: the client starts a new attempt, and the raw files it already uploaded are reused by hash.
  • The same sweeper turns a silent feed (a schedule, SyncSchedule, passed by more than its grace with no SyncRun) into an issue for the Organization and for support, and frees ParseJobs whose lease expired (ParseJob.lease_until).
  • Diff: received keys and hashes are compared with the stored evidence of the feed in the coverage.
  • New key = added; changed content appends a revision; removal appends a cancellation revision.
  • Infer removals only from verified full snapshots with identical resource/filter scope and authoritative source ordering.
  • Missing pages, parser rejects, changed filters, outages and older snapshots never cancel current evidence.
  • Incremental feeds delete only by explicit tombstone; unknown snapshot ordering blocks inferred deletions.
  • Apply validated changes atomically per feed/scope, then re-resolve; repeated run must be idempotent.
  • Change detection never relies on our updated_at (it is our write time) nor only on the source's modified time (often missing).
  • After the diff only affected animals/groups/material and their dependent resolutions/periods are recomputed.
  • What a sync changed, record by record, is read from the data itself: added = a revision 1, changed = a later revision, removed = a cancellation revision, each with available_in pointing to a write (WriteCommit) of this sync_run; registry rows (animals, identifiers, ...) keep their before/after images in SystemVersion with the same write numbers.