4.7 Ingestion

Ingestion is how farm data reaches us: milking robots, herd software, labs and registries produce files at the farm or in a vendor's cloud, and the ingestion components of farm (its ingestion API, vendor pullers and parser workers, the Sources, Files and Sync domains of its interface) turn them into the canonical data it keeps. Ingestion was a system of its own until 2026-10-30; it is one system with the canonical data because the two are written in the same transactions. It must survive the same file arriving twice, a sync cut halfway, a parser with a bug and a farm with a slow line, without ever losing or duplicating a value. This page follows one sync from the farm to the database and explains where each piece of data is kept.

The farm sync client extracts by itself, and nothing else (decided 2026-11-10). Each integration reads a vendor's software on the farm PC (its database, its export, its API) with the extractor of a strategy, on the schedule, with no action from the farmer once it is set up. There is no folder where a person drops files and no manual upload from the client: data a person hands us by hand is not farm software data, and it would bypass the detection of the source and its version.

• The generic folder reader (moonitoring.generic_raw) exists only in local and test builds of the client, to try the ingestion path without a vendor; stage and prod builds do not contain it.
• An integration appears in the client only when a strategy for that software exists; until the first vendor strategies are built, a stage or prod client has nothing to read and says so.
• Integrations are set up in the sync client, by a signed-in person (decided 2026-11-10) who holds farm.sources.manage on the farm, with a sign-in at most five minutes old; otherwise the client opens the browser again with prompt=login. The person picks the software from the client's catalogue of supported software (only software with a strategy is listed: the client-side adapter mappings that apply to the farm, global or of its Organization, which the client receives with its sync configuration even before it has a feed), fills the set-up form of that software, runs Test connection, which the client does on the computer, and the integration becomes a card of the client. The client then connects the feed for itself (Sources).
• What the client needs to read the software stays on the computer (decided 2026-11-10): a database account or a server password is kept in the Windows Credential Manager of the client and never sent to us.
• Cloud vendor integrations are set up in the client too (decided 2026-11-10), such as the milk recording of AIA, which farm pulls: the same catalogue and the same fresh sign-in. The client creates the feed and sends the credential once with PUT /feeds/{feed_id}/credential; it goes to Secret Manager, is never returned, and the vendor puller of farm uses it. The client keeps no copy. It offers only the cloud vendors for which farm runs a puller (an adapter of side puller mapped to that channel, listed in its sync configuration), and shows the farm's cloud vendor feeds as read-only cards.
• The Organization Dashboard and the Console show the integrations and their problems (feeds, runs, integration gaps) and set up nothing. The Dashboard pauses an integration through its schedule, since schedules are set there (confirmed 2026-11-10).

4.7.1 The Path of a Sync

sequenceDiagram participant farm as Farm sync client participant api as farm ingestion API participant raw as raw bucket participant db as Cloud SQL (farm) participant ps as Pub/Sub participant w as Parser workers farm->>farm: 1 · Read and classify files, compress, hash farm->>api: 2 · Which of these hashes do you hold? api-->>farm: 3 · Only the new ones, with signed upload URLs farm->>raw: 4 · Resumable upload, straight to the bucket farm->>api: 5 · Commit the sync api->>db: 6 · SyncRun, RawMetadata, ParseJobs in one transaction db->>ps: 7 · Outbox relay publishes parse.requested (ordered per feed) ps->>w: 8 · Deliver the job w->>raw: 9 · Read the raw file w->>db: 10 · Write canonical revisions under one WriteCommit

1. The farm sync client reads the files of each source, recognizes the feed and its version, compresses each file and computes the hash of its uncompressed content.
2. It asks the ingestion API of farm which hashes we already hold, and uploads only the others, through short-lived signed URLs, directly to the raw bucket. The API never sits in the path of the bytes, so a large upload costs it nothing.
3. Once the files are uploaded, the client commits the sync: one transaction records the SyncRun, one RawMetadata per new file and one ParseJob per file to parse.
4. A relay publishes the jobs to Pub/Sub, with the feed as ordering key: the files of one feed are parsed in order, all the feeds of all farms in parallel.
5. A parser worker reads the raw file, compares it with what we already hold, and writes canonical revisions under one WriteCommit. Values it cannot represent are recorded as skips, never approximated.

Vendor pullers, which fetch data from vendor clouds on a schedule, enter the same path at step 3: they store the file in the bucket and commit it like a farm client.

4.7.2 Four Gates Against Duplicates

A file or a record can arrive twice for many reasons: a retried upload, a full snapshot that repeats last week, a message delivered twice. Four gates make each repetition harmless:

Gate	Where	Stops
1 · Known hash	Before upload	A file the feed already delivered is neither uploaded, stored nor parsed again
2 · One active job per file	When a job is created	Two parses of the same file running at once
3 · Job ID with a lease	When a worker takes a job	A message delivered twice, or a worker that died halfway and is replaced
4 · Unchanged record	When writing	A record identical to the last revision produces no new revision

4.7.3 When Something Fails

• A failed job is retried with growing delays. After a set number of attempts it moves to parse.dead, where it waits for a person; the other feeds keep flowing, and only the stuck feed is held.
• Failed jobs and runs are listed in the Console. From there a staff member can ask for a re-parse of one file, one run, or one feed over a period, for example after fixing a parser. A re-parse is a new set of jobs on the same queue, through the same gates, so it only writes what changed.

4.7.4 Where Data Is Kept

What	Where	Why there
Metadata of every sync, file and job: SyncRun, RawMetadata, ParseJob, skip counts	Cloud SQL, farm database	Queried and joined constantly; small
Canonical data: observations, events, periods, with their revisions	Cloud SQL, farm database, partitioned by farm and month	What the other systems read, at any write number
Raw files, compressed	raw bucket	Written once, read to parse and re-parse; the only way to rebuild canonical data
Row-level skip detail, JSONL next to each raw file	raw bucket	Evidence for every skipped value, too bulky for the database
Datasets for models, Parquet	datasets bucket	Read by training and BigQuery in place

Raw Files

• Compressed by the client before upload, and stored under the hash of their uncompressed content, so the same data compressed differently is still recognized.
• Kept for as long as any canonical record derives from them; they leave the bucket only through an erasure request.
• Bucket: moonitoring-<env>-farm-raw, EU dual-region (Milan and Belgium). Raw files cannot be produced again, so they are kept in two regions.
• Storage class: written in Standard, moved to Archive by a lifecycle rule after 30 days.

The first 30 days cover parsing and the re-parses that follow a parser fix, which come early. After that a raw file is read only by a rare re-parse, and Archive costs about a tenth of Standard to keep. Archive in Google Cloud answers in milliseconds, like Standard; what it adds is a fee for each read and a minimum of 365 days of storage per file, which an immutable file kept for years always meets.

Why not Autoclass

Autoclass moves files between classes by itself, but charges a monthly fee per thousand files and reaches Archive only after a year without access. Our access pattern is known in advance, so a fixed lifecycle rule is simpler and cheaper.

At our scale the class makes little difference: about 1 MB of compressed raw data per farm per day is 36 GB a year at 100 farms, a few dollars either way. The rule matters as the archive grows over the years.

4.7.5 Capacity

Cloud Run, Pub/Sub and Cloud Storage scale far beyond our needs; the limit is the Cloud SQL instance. With prod's current instance (1 vCPU, 3.75 GB, highly available), about 100 farms, a million observations a day, fit comfortably, and up to about 250 to 300 when farm syncs are spread over the night by their schedules. This is an estimate from the model, not a measurement.

Not decided yet: how long canonical data stays in Cloud SQL

Canonical data grows by about 1.1 GB per farm per year, and Cloud SQL storage with high availability costs about twenty times Cloud Storage. Moving partitions older than 6 to 12 months to Parquet files in Cloud Storage would divide that line by 5 to 10, but reads "at number N" on old data would then go through BigQuery. A Cloud SQL disk never shrinks, so this must be decided before the database grows.