Skip to content

6.9.1 Introduction

Versioning & Reproducibility makes it possible to see the data exactly as it was at any past moment. Every change gets a sequential write number and records keep their earlier versions, so a model run or a price can be computed again from exactly the same data.

6.9.1.1 Model notes

One clock: the write number

A model trained or a price computed months ago must be recomputable exactly, even after the farm corrected data at the source. One clock for everything: the write number (WriteCommit).

  • Evidence and decisions (observations, sessions, events, samples, milk material, resolutions, evidence links, animal links, selections, periods, animal profile evidence, feed coverage) are revision chains (RecordRevision): a correction, cancellation or changed decision appends a revision and never updates the old payload.
  • Registry rows (animals, identifiers, group memberships, ownerships, breed memberships, device assignments) keep a stable id; every change keeps the previous image with its write-number interval (SystemVersion).
  • The data as known at write number N (N at or below the watermark, WriteCommit): for each logical record, the revision with available_in ≤ N and (replaced_in empty or > N); for registry rows, the image with created_seq ≤ N and (retired_seq empty or > N).
  • A model run records the write number it read, the feature code version, the rule set and the clock (ModelRun): for history imported later, "what we knew at N" is not "what the farm knew then"; the source clock (recorded_at) answers that.

Features for models are not stored as canonical rows; they are computed by versioned code from the data at N.

Datasets

  • Datasets are snapshots. A data scientist defines a dataset in the Console (MlDataset): which tables, which time ranges, animal types, breeds, origins, nations, Organizations. Building it reads the data at a write number and writes Parquet files to the datasets bucket, one immutable version (MlDatasetVersion); the definition can be built again later at a newer number, and every version stays readable as it was. Files are addressed by their content (MlDatasetFile), so versions and datasets that select the same partitions share the same files, and a file is removed only when no version references it. A notebook or a pipeline reads a version by its manifest path, never the database. Data for notebooks is pseudonymized at build time; only scoring and pricing runs in production read identified data.
  • A variant (a narrower selection of an existing version) is requested from the Console or from a notebook through the ai-model-platform admin API and built from the parent's files on the server: nothing is downloaded or uploaded by the data scientist.

6.9.1.2 Diagram

Entities of this domain, their subtypes and the references between them. References to other domains are listed on each entity page.

classDiagram class WriteCommit class ModelRun class ErasureRequest class MlDataset class MlDatasetVersion class MlDatasetFile class Model class ModelVersion class ScoringBatch ModelRun --> WriteCommit : data_seq ModelRun --> ModelRun : previous_run ModelRun --> MlDatasetVersion : dataset_version ErasureRequest --> WriteCommit : executed_in MlDatasetVersion --> MlDataset : dataset MlDatasetVersion --> WriteCommit : data_seq MlDatasetVersion --> MlDatasetFile : files MlDatasetVersion --> MlDatasetVersion : parent_version Model --> ModelVersion : deployed_version ModelVersion --> Model : model ModelVersion --> ModelRun : training_run ScoringBatch --> ModelVersion : model_version ScoringBatch --> WriteCommit : data_seq ScoringBatch --> ModelRun : model_run MlDataset --> WriteCommit : system_version

6.9.1.3 Entities

  • WriteCommit: One saved change to versioned data, with its write number.
  • ModelRun: One reproducible use of the data by a model: training, scoring, pricing or an analysis.
  • ErasureRequest: A farm that leaves, or asks for its data to be removed.
  • MlDataset: A named, reusable selection of canonical data for model work, created and managed by data scientists in the Console.
  • MlDatasetVersion (immutable): One build of a dataset: the selected data as it was at a write number, written to Parquet files in the datasets bucket and described by a manifest.
  • MlDatasetFile (immutable): One Parquet file of one partition of one table, named by the hash of its content.
  • Model: A model other systems can call: its name, what it predicts and the entitlement feature an Organization needs to use it.
  • ModelVersion: One trained version of a model, produced by a training run and kept as an artifact in the model registry.
  • ScoringBatch: One batch scoring: a deployed model version reads the farm data of a scope at a write number, records a run and appends its predictions to farm as computed records, one write per farm.