How the numbers get made
The operational side of the same dataset: what the pipeline does on a given day, which rows it refused, and which thresholds it was judged against. Every figure here comes from a manifest the pipeline wrote next to its own output.
A green run that quietly processed nothing is the failure mode worth catching, so the checks below run on days with no work too.
The pipeline
daily schedule · work decided by ETag
The source publishes monthly with a 26–85 day lag and rewrites files after the fact — on 2026-03-25 it replaced December, January and February inside one nine-minute window. So the daily run does not process "yesterday". It issues six HTTP HEAD requests, compares ETags against what it already stored, and usually does nothing at all.
_meta.json stored beside the raw file.
How it runs
The same container image in both runners. Business logic lives in the application, not in the orchestrator.
Quality thresholds
Derived from a measured baseline over the full history, not guessed. Every run records the thresholds that applied to it.
What happens when a run fails
Three failures look identical from the outside — a red task. Only one of them is worth repeating.
| Error | Cause | Behaviour |
|---|---|---|
| TransientError | 5xx, timeout, connection reset | Retry twice with exponential backoff |
| PermanentError | 403/404 — the month is not published yet | Fail fast. detect never hands such a month to the mapper in the first place |
| DataQualityError | A run threshold was exceeded | Fail fast — a retry downloads the same bytes and fails the same way |
A blanket retries=3 would mean an unpublished month takes 45 minutes to die
and broken data gets computed four times. Alerts fire only once retries are exhausted, and
skipped tasks send nothing — otherwise everyone filters the channel within a week and
misses the one real incident.
Runs
click a row to load that month below
One row is one _runs/<run_id>.json manifest sitting next to the output.
A manifest is written only after a successful write, so this table lists runs that finished
— failed executions live in Step Functions history, not here. Partitions get overwritten,
manifests accumulate, so "how many times was this month recomputed, and against which source
ETag" is a question you answer with ls rather than an incident review.
| Partition | Trigger | Run | Input rows | Published | Quarantined | Output | Time | Source ETag |
|---|
What the rules caught
Where the rows went
Quality rules
How many rows each rule touched. Counted independently — one row can break several rules at once, so these do not add up to the quarantine count above.
Rejected rows keep their reason
Nothing is deleted. What a rule refuses lands in a rejects Parquet next to the output.
A row is quarantined only when no positive spend follows from it. A bad value is nulled and left out of its own average, but neither the row nor its money is thrown away: dropping whole rows would have cost $2.32M of revenue and 88,743 trips while moving the mean distance by 0.1 %. Every average in the output therefore ships with its own denominator, so "measured on how many trips" is answerable per zone and per day rather than as one global footnote.
The output itself — 263 pickup zones on a map, eight metrics, daily series per month, and the mean sitting next to the median it should be read against.
◂ The data