Skip to content

6.9.2.5 MlDatasetVersion

Immutable

Records are never updated after creation.

6.9.2.5.1 Purpose

One build of a dataset: the selected data as it was at a write number, written to Parquet files in the datasets bucket and described by a manifest. A version is never changed after it is ready; a notebook or a pipeline opens it by its manifest path and gets the same rows forever, which is what makes a ModelRun reproducible. Deleting a version removes only the files no other version shares.

6.9.2.5.2 Questions it answers

  • Which write number, rule set and export code produced these files?
  • Which files make up the version, with what hash, size and row count?
  • Is this version identical to another one (same manifest hash)?
  • Which version was this variant derived from?
  • Until when is the version kept, and may it be deleted at all?

6.9.2.5.3 Fields

Field Type Cardinality Description
id string (uuid) 1..1 Key.
dataset MlDataset 1..1
number integer 1..1 1, 2, 3 within the dataset.
data_seq WriteCommit 1..1 The write number read; at or below the watermark.
rule_set ReconciliationRuleSet 1..1
knowledge_clock enum 1..1 Values: platform, source.
As 62.12.
selection MlDatasetSelection 1..1
feature_code_version string 1..1
anonymization enum 1..1 Values: identified, pseudonymized, anonymized.
manifest_path string 1..1 Gs://<datasets bucket>/<dataset slug>/v<number>/manifest.json: schema per table, the list of files with their hashes and partitions, row counts, selection, data_seq, knowledge_clock, rule_set, feature_code_version and anonymization configuration. What a notebook opens.
manifest_hash string 1..1 SHA-256 of the manifest: the identity of the version and the dataset_hash (ModelRun.dataset_hash) of every run that reads it.
files MlDatasetFile 0..*
row_count integer 1..1
size_bytes integer 1..1
status enum 1..1 Values: building, ready, failed, deleted. Default: building.
built_by MoonitoringUser 1..1
built_at datetime 0..1
keep_until datetime 0..1 Set from the dataset's retention_days; never set while a ModelRun of a model in production references the version.
deleted_at datetime 0..1
parent_version MlDatasetVersion 0..1 Set when this version is a variant derived from another one (same data_seq, narrower selection): it is built from the parent's files, never from the database, and shares every file the narrower selection keeps whole.
created_at datetime 1..1

6.9.2.5.4 Relationships

References

Referenced by

Uses types

MlDatasetSelection

6.9.2.5.5 Rules & constraints

  • One build of a dataset: the selected data at a write number, as files, plus a manifest. Never changed after it is ready; deleting it removes the files no other version shares.
  • (dataset, number) is unique.
  • Reuse compares data_seq, knowledge_clock, selection, feature_code_version, rule_set and anonymization configuration.
  • Different knowledge clocks cannot reuse the same version merely because the other inputs match.
  • The manifest records these inputs; an identical build reuses the existing version.