Skip to content

6.4.2.13 RawMetadata

6.4.2.13.1 Purpose

Points to one original file, kept in file storage exactly as we received it, with a fingerprint (hash) and size to check that it is intact. Values in raw files never enter canonical data directly: parsers read the file again whenever a mapping changes.

6.4.2.13.2 Questions it answers

  • Where is the original file behind a record?
  • Which producer sent it, and is the file still present?
  • Is the stored file intact?

6.4.2.13.3 Fields

Field Type Cardinality Description
id string (uuid) 1..1 Key.
source_producer SourceProducerEntity 1..1 The producer of the raw data.
data_path string 1..1 Path to the raw compressed data file in the storage system.
data_hash string 1..1 Hash of the uncompressed content, so the same data compressed differently is still recognised.
hash_type enum (HashEnum) 1..1 Values: xxh3, sha256, md5.
data_size integer 1..1 Size of the raw uncompressed data file in bytes.
deleted_at datetime 0..1 When the raw data file was deleted from the storage system; empty when still present.
source_feed SourceFeed 1..1 The feed of source_producer (SourceProducerEntity.source_feed), repeated here to make a file unique per feed.
created_at datetime 1..1
updated_at datetime 1..1

6.4.2.13.4 Relationships

References

Referenced by

6.4.2.13.5 Rules & constraints

  • (source_feed, hash_type, data_hash) is unique: a file the feed already delivered is not stored or parsed again.
  • A feed uses one hash_type, so the same content always gets the same key. Before uploading, the client asks which hashes we already hold and sends only the others; a SyncRun that delivers a known file lists the existing RawMetadata in its SyncRun.raw_inputs, which keeps the source_producer and received_at of the first delivery.
  • Raw data is re-parsed from here when a parser mapping changes (e.g. a unit is confirmed) or when a Moonitoring user asks for it (ParseJob).
  • Storage: the raw bucket is EU dual-region; files are written in Standard and a lifecycle rule moves them to Archive after 30 days, when parsing and early re-parses are done. No Autoclass: it charges per object and reaches Archive only after a year without access.