Skip to content

6.5.1 Introduction

Observations holds the values measured or computed about animals, groups and farm products, such as a milk yield or a lab result. Each observation has a versioned type that defines what it means and a context that says how it was obtained; sessions and samples group the values produced together. The model notes below define canonical data, our standard form, for observations, events and periods alike.

6.5.1.1 Model notes

Canonical values

  • Canonical = defined semantics in our convention, not merely a converted number. Raw values from providers never enter these tables: they stay in the bucket, referenced by RawMetadata. Vendor conventions live in the parser mappings.
  • A value we cannot represent with a defined ObservationType, unit and subject is not published: it is skipped and logged (IngestionSkip).
  • So a missing CanonicalObservation means "not measured or not publishable": check IngestionSkip before assuming not measured.
  • Missing, censored or invalid results are never replaced with zero. Zero is stored only when it is a real zero.

What is canonical

  • Raw signals (e.g. accelerometer samples) are raw. The quantities and facts produced from them are canonical (e.g. rumination minutes, steps, milk kg, a heat alert as a HeatEvent), whether we compute them from raw data or they arrive ready from the device or the management software and only need converting.
  • Proprietary indices, scores and probabilities of other vendors, which we cannot represent with a defined meaning and unit, are not canonical: they stay only in the raw file (IngestionSkip reason proprietary_index).
  • Values the vendor derives from a series we already ingest (expected values, rolling averages, % deviations) are not ingested; we recompute them.

Parser rules

Validated on real rows from Lely, AfiMilk, DelPro, DairyComp, SenseHub, Allflex and lab tests.

  • State fields repeated on every export (lactation status, current group, last calving date) are not observations: parsers derive events and periods from them. An inferred boundary is not an observed instant.
  • A measurement written inside an event (e.g. a ketone value in an event note) becomes a CanonicalObservation.
  • Source labels are mapped by documented meaning and export version, not by label alone.
  • Every record carries a source_record_key, so re-sent or re-exported data is recognised and not duplicated.
  • The mapping sets the origin (SourceOrigin) per source field, and per row when the source marks how the value was obtained (e.g. DelPro ManualMode, ICAR milkingType Manual/Automated): a manual row gets an OperatorSourceOrigin even inside a sensor table. When the source does not say, the origin stays UnknownSourceOrigin; it is never inferred from the channel.
  • Facts that become known later (a later outcome, a planned removal) are stored at the write number at which they arrived, never backdated: a model reading at an earlier write number cannot see them (WriteCommit, ModelRun).

  • Identity, deduplication, revision and cross-source preference are different operations.

  • STORAGE. Every canonical record (observation, session, event, period, profile evidence) carries farm: the farm of the feed that delivered it. It is the partition key, the key of row-level access and the scope of erasure (ErasureRequest); access is still decided by Ownership, farm only makes the check cheap. Observations and sessions are partitioned by farm and by month of their time (about 1M observation rows a day at 100 farms); events, periods and profile evidence by farm. Partitioning is a storage choice: the logical model does not change.

  • Retention: raw files are kept for as long as any canonical record derived from them exists, because they are the only way to re-parse; they leave the bucket only through erasure (ErasureRequest). Cloud Logging keeps 30 days, so every fact needed to explain data or a sync lives in the database or in the bucket, never only in the logs.

Who can change canonical data

  • Data is corrected where it was produced. Data from labs, sensors, management software and registries is never edited in our app: the farm, or whoever is responsible for that source, fixes it at the source and the next sync brings the change.
  • In our app, members of the owning Organization with capability farm.animal_records.edit_own can edit or cancel only the data that entered through that Organization's own feeds (MoonitoringAppChannel entries, FileChannel uploads). Removing an uploaded file cancels every record parsed from it. Each edit is a WriteCommit of kind app_edit (WriteCommit).

6.5.1.2 Diagram

Entities of this domain, their subtypes and the references between them. References to other domains are listed on each entity page.

classDiagram class CanonicalObservation class AnimalObservation class AnimalGroupObservation class ProductObservation class ObservationType class ObservationMethod class ObservationContext class BodySite class ObservationSession class Sample class ProductMaterial class MilkMaterial class EggLot class ObservationSelection class CanonicalDerivation CanonicalObservation <|-- AnimalObservation CanonicalObservation <|-- AnimalGroupObservation CanonicalObservation <|-- ProductObservation ProductMaterial <|-- MilkMaterial ProductMaterial <|-- EggLot CanonicalObservation --> ObservationType : observation_type CanonicalObservation --> BodySite : body_site CanonicalObservation --> ObservationSession : session CanonicalObservation --> Sample : sample CanonicalObservation --> CanonicalDerivation : derivation CanonicalObservation --> ObservationContext : context ProductObservation --> ProductMaterial : product_material ObservationType --> BodySite : allowed_body_sites ObservationContext --> ObservationMethod : method ObservationSession --> ProductMaterial : product_material Sample --> ProductMaterial : product_material Sample --> ObservationSession : collection_session Sample --> ObservationSession : contributing_sessions Sample --> BodySite : body_site ObservationSelection --> CanonicalObservation : candidates ObservationSelection --> CanonicalObservation : preferred

6.5.1.3 Entities

  • CanonicalObservation: One measured or computed value about an animal, a group or a product lot, in our standard form: what was observed (an observation type), when, how (an observation context), on which body site, and with which value.
  • ObservationType: The versioned catalog entry that defines what an observation means: the property, material, result shape, canonical unit and validation.
  • ObservationMethod: A versioned analytical, measurement or computation protocol.
  • ObservationContext: How a value was obtained: method, result basis, time basis and aggregation.
  • BodySite: Catalog of the parts of a subject that a value can refer to, such as an udder quarter or a leg.
  • ObservationSession: A bounded activity on a subject, optionally with a device: a milking, weighing, feeding or sampling.
  • Sample: A physical specimen, such as a milk sample sent to a lab.
  • ProductMaterial: A physical lot of farm product, such as bulk-tank milk.
  • ObservationSelection: Chooses one preferred copy among observations that are proved to be copies of the same original measurement, for example a reading received both directly and through the management software.
  • CanonicalDerivation: Records how we computed a value from other values: the calculation, its parameters and the exact versions of its inputs.