Skip to content

6.4.1 Introduction

Provenance & Ingestion explains where every value comes from and how it gets into the platform. Origin, channel and device say who produced a value and how it reached us. Feeds, raw files, parse jobs and skips record how received files became canonical data, and feed coverage says how complete that data is. Adapters are the readers the farm sync client and the parsers choose by source and version; what neither can read yet becomes one integration gap, however many farms meet it.

6.4.1.1 Model notes

Origin, channel and device

Every canonical record answers three independent questions:

  • Origin (SourceOrigin): who or what produced the value (a kind of sensor, a person, a lab, a computation).
  • Channel (SourceChannel): how the value reached us (direct from the device, via management software, via a third-party platform, via our app, via a file).
  • Device (Device): which physical device produced it, when known.

Origin and channel combine freely: the same robot reading can arrive directly or through the management software.

Lineage

From any canonical record (observation, session, event, period, profile evidence) we can always reconstruct, with stored references only:

  • The raw data: raw_metadata (RawMetadata, the file in the bucket) + raw_locator (record/field in it) + source_record_key;.
  • The source: source_producer (SourceProducerEntity) → origin_version → SourceOrigin (who, assurance) and channel_version → SourceChannel (how), source_feed (SourceFeed: farm, channel, upstream_editable, reliability_class), device (Device); the raw file carries its own source_producer, so raw and canonical answer the same question the same way;.
  • The write: revision.available_in → WriteCommit: parse job and parser/mapping version, sync run, app user;.
  • For records we compute: ComputedSourceOrigin + its OriginVersion (the algorithm), and for observations and periods the exact inputs ((derivation, CanonicalDerivation)) → their own lineage.

raw_metadata is required on every record except records we compute. Data typed in our app is no exception: the submitted payload is stored as a raw file of the feed's MoonitoringAppChannel.

How reliable a value is, is never stored as one score: it is read from origin + assurance + channel + feed custody.

6.4.1.2 Diagram

Entities of this domain, their subtypes and the references between them. References to other domains are listed on each entity page.

classDiagram class SourceProducerEntity class SourceVersion class OriginVersion class ChannelVersion class SourceOrigin class SensorSourceOrigin class OperatorSourceOrigin class LabSourceOrigin class ComputedSourceOrigin class UnknownSourceOrigin class SourceChannel class DirectDeviceChannel class ManagementSoftwareChannel class ThirdPartyPlatformChannel class MoonitoringAppChannel class FileChannel class Device class DeviceAssignment class SourceFeed class SourceAdapter class AdapterMapping class FeedDetection class IntegrationGap class IntegrationGapOccurrence class RawMetadata class IngestionSkip class ParseJob class FeedCoverage SourceVersion <|-- OriginVersion SourceVersion <|-- ChannelVersion SourceOrigin <|-- SensorSourceOrigin SourceOrigin <|-- OperatorSourceOrigin SourceOrigin <|-- LabSourceOrigin SourceOrigin <|-- ComputedSourceOrigin SourceOrigin <|-- UnknownSourceOrigin SourceChannel <|-- DirectDeviceChannel SourceChannel <|-- ManagementSoftwareChannel SourceChannel <|-- ThirdPartyPlatformChannel SourceChannel <|-- MoonitoringAppChannel SourceChannel <|-- FileChannel SourceProducerEntity --> OriginVersion : origin_version SourceProducerEntity --> ChannelVersion : channel_version SourceProducerEntity --> SourceFeed : source_feed OriginVersion --> SourceOrigin : source_origin ChannelVersion --> SourceChannel : source_channel Device --> SensorSourceOrigin : source_origin DeviceAssignment --> Device : device DeviceAssignment --> SourceProducerEntity : source_producer SourceFeed --> SourceChannel : source_channel SourceAdapter --> SourceChannel : source_channel FeedDetection --> SourceFeed : source_feed FeedDetection --> SourceChannel : source_channel AdapterMapping --> SourceChannel : source_channel AdapterMapping --> SourceAdapter : adapter AdapterMapping --> SourceFeed : source_feed IntegrationGap --> SourceChannel : source_channel IntegrationGap --> AdapterMapping : resolved_by IntegrationGapOccurrence --> IntegrationGap : gap IntegrationGapOccurrence --> SourceFeed : source_feed IntegrationGapOccurrence --> ParseJob : parse_job IntegrationGapOccurrence --> RawMetadata : sample RawMetadata --> SourceProducerEntity : source_producer RawMetadata --> SourceFeed : source_feed IngestionSkip --> RawMetadata : raw_metadata IngestionSkip --> ParseJob : parse_job ParseJob --> RawMetadata : raw_metadata ParseJob --> ParseJob : previous_job FeedCoverage --> SourceFeed : source_feed

6.4.1.3 Entities

  • SourceProducerEntity: Says where a piece of data comes from: which origin version produced it, through which channel version and feed it arrived, when it was produced and when we received it.
  • SourceVersion: A version of an origin (sensor firmware, lab method, algorithm) or of a channel (a management-software release).
  • SourceOrigin: Who or what produces values: a kind of sensor, a human operator, a lab, a computation or an unknown producer.
  • SourceChannel: How values reach us: read directly from a device, through management software, through a third-party platform, typed in our app or uploaded as a file.
  • Device: A physical device on a farm: a specific robot, meter, collar, tag, scale or camera.
  • DeviceAssignment: Records which animal wears or carries a mobile device (a collar, tag or bolus) over time, so its readings are attributed to the right animal.
  • SourceFeed: One farm's data arriving through one channel, for example a farm's data from its management software.
  • SourceAdapter: A reader of one kind of source, for a range of its versions: the strategy the farm sync client or a parser uses for a given source.
  • AdapterMapping: Which adapter reads which detected source version, chosen by staff in the Console instead of built into the client.
  • FeedDetection
  • IntegrationGap: Something we cannot read yet, recorded once however many farms meet it: a source version no client adapter covers, or a file structure no parser understands.
  • IntegrationGapOccurrence: One meeting of an integration gap by one farm, through a failed sync run or a failed parse job.
  • RawMetadata: Points to one original file, kept in file storage exactly as we received it, with a fingerprint (hash) and size to check that it is intact.
  • IngestionSkip: Counts the source values that were not published as canonical data, per parse job, field and reason, with a row-by-row log next to the raw file.
  • ParseJob: The work queue of the parsers, the programs that read raw files: one job per raw file and parser version.
  • FeedCoverage: Says which period of a feed's data is present, complete and final, for each kind of data.