Skip to content

6.9.2.6 MlDatasetFile

Immutable

Records are never updated after creation.

6.9.2.6.1 Purpose

One Parquet file of one partition of one table, named by the hash of its content. Because the name is the content, two versions, or two datasets, whose builds yield the same partition point at the same object and store it once; a new version of a growing dataset adds only the partitions that changed. A file is removed when no version references it.

6.9.2.6.2 Questions it answers

  • Which versions share this file?
  • Which table, partition and anonymization level does the file hold?
  • How much of the bucket does a dataset really add, net of shared files?

6.9.2.6.3 Fields

Field Type Cardinality Description
hash string 1..1 Key.
SHA-256 of the file content; also the object name in the bucket.
path string 1..1 Gs://<datasets bucket>/files/<hash>.parquet.
table enum 1..1 Values: animals, observations, events, resolved_events, periods, animal_links.
partition string (JSON) 1..1 The partition keys of this file, e.g. {"month": "2025-03", "animal_type": "bovine"}. Never a farm or animal id.
anonymization enum 1..1 Values: identified, pseudonymized, anonymized.
Files of different levels never share content; a rotated pseudonymization key makes new files.
row_count integer 1..1
size_bytes integer 1..1
created_at datetime 1..1

6.9.2.6.4 Relationships

Referenced by

6.9.2.6.5 Rules & constraints

  • One Parquet file, one partition of one table, addressed by its content: the object name is its hash. Shared by every version whose build yields the same partition, so two datasets over the same months store it once. Removed when no version references it.
  • Parquet with ZSTD compression, dictionary encoding, row groups of about 100 MB, rows sorted by animal and time inside a file; a partition is one month of one table for one animal type, split into more files above 512 MB.