Skip to content

6.4.2.15 ParseJob

6.4.2.15.1 Purpose

The work queue of the parsers, the programs that read raw files: one job per raw file and parser version. It makes receiving the same file twice harmless and records how raw data became canonical data. A failed job carries a failure class that decides whether it is retried or waits for a new parser; it never changes the status of the SyncRun that delivered the file.

6.4.2.15.2 Questions it answers

  • Has this raw file been parsed, and with which parser version?
  • Which jobs failed or are waiting for a retry?
  • Was a failure transient (retry) or permanent (a new parser or a manual job)?
  • Which job wrote a set of records?

6.4.2.15.3 Fields

Field Type Cardinality Description
id string (uuid) 1..1 Key.
raw_metadata RawMetadata 1..1
parser_version string 1..1
status enum 1..1 Values: pending, running, done, failed.
attempts integer 1..1 Default: 0.
next_attempt_at datetime 0..1
lease_until datetime 0..1 A running job whose lease expired is taken again by the scheduler.
last_error string 0..1
trigger enum 1..1 Values: upload, parser_change, manual.
upload: first parse of a new raw file.
parser_change: a new parser or mapping version.
manual: asked by a Moonitoring user, e.g. to retry a failed job or to re-parse after an animal identity was resolved.
requested_by MoonitoringUser 0..1 Who asked; set only when trigger = manual.
reason string 0..1 Why, in one line a person can act on, e.g. "unit of field DIM confirmed"; set only when trigger = manual.
previous_job ParseJob 0..1 The job this one runs again, for a retry or a re-parse of the same raw file.
failure_class enum 0..1 Values: timeout, lease_expired, dependency_unavailable, invalid_file, unsupported_version, parser_error.
Set only when status = failed. Transient classes (timeout, lease_expired, dependency_unavailable) are retried with backoff until attempts reach the limit, then the message goes to the dead-letter topic. Permanent classes (invalid_file, unsupported_version, parser_error) are not retried: the job fails at once and waits for a new parser version (trigger parser_change) or a manual job. last_error says why.
created_at datetime 1..1
updated_at datetime 1..1

6.4.2.15.4 Relationships

References

Referenced by

6.4.2.15.5 Rules & constraints

  • Work queue for parsers (outbox pattern). Pub/Sub may deliver twice; the job and the source keys make it harmless.
  • A failed job never changes the status of its SyncRun: the run says whether the client delivered, the jobs say whether we processed what it delivered.
  • (raw_metadata, parser_version) is unique among jobs with status pending or running: the same work is never queued twice at once, but a finished or failed job can be run again as a new job.
  • A re-parse is asked for one raw file, for every raw_input of a SyncRun (done or failed), or for a feed and period; each raw file gets its own job. Records whose source_record_key and content are unchanged produce no new revision, so a re-parse writes only what changed.