NYC Taxi Pipeline
Every number on this page comes from manifests and Parquet files the pipeline actually wrote to S3. Nothing is mocked, sampled, or rounded up for effect.
The page is static. The whole data layer is baked into the file — no API, no database, nothing running between the moments someone looks at it.
The pipeline
daily schedule · work decided by ETag
The source publishes monthly with a 26–85 day lag and rewrites files after the fact — on 2026-03-25 it replaced December, January and February inside one nine-minute window. So the daily run does not process "yesterday". It issues six HTTP HEAD requests, compares ETags against what it already stored, and usually does nothing at all.
_meta.json stored beside the raw file.
How it runs
The same container image in both runners. Business logic lives in the application, not in the orchestrator.
Quality thresholds
Derived from a measured baseline over the full history, not guessed. Every run records the thresholds that applied to it.
What happens when a run fails
Three failures look identical from the outside — a red task. Only one of them is worth repeating.
| Error | Cause | Behaviour |
|---|---|---|
| TransientError | 5xx, timeout, connection reset | Retry twice with exponential backoff |
| PermanentError | 403/404 — the month is not published yet | Fail fast. detect never hands such a month to the mapper in the first place |
| DataQualityError | A run threshold was exceeded | Fail fast — a retry downloads the same bytes and fails the same way |
A blanket retries=3 would mean an unpublished month takes 45 minutes to die
and broken data gets computed four times. Alerts fire only once retries are exhausted, and
skipped tasks send nothing — otherwise everyone filters the channel within a week and
misses the one real incident.
Runs
click a row to load that month below
One row is one _runs/<run_id>.json manifest sitting next to the output.
A manifest is written only after a successful write, so this table lists runs that finished
— failed executions live in Step Functions history, not here. Partitions get overwritten,
manifests accumulate, so "how many times was this month recomputed, and against which source
ETag" is a question you answer with ls rather than an incident review.
| Partition | Trigger | Run | Input rows | Published | Quarantined | Output | Time | Source ETag |
|---|
The map
Every trip carries the pickup zone it started in, so the curated layer is already a map —
it just needs the outlines. These come from the TLC zone shapefile, simplified once into
web/zones.json; the build bakes them in like everything else. Hover a zone
to read its numbers.
Six buckets holding the same number of zones each, so the colour is a ranking, not a ratio — equal-width bands would paint everything except Midtown the same shade. Means built on a handful of trips are noisy, so the tooltip always shows the trip count underneath them.
Data
Every month in curated
Published trips per source month. Click a bar to load it.
Where the rows went
Trips per day
Shaded bands are weekends. Saturday and Sunday have their own profile, so without them the Monday drop reads as an incident.
Show as a table
Quality rules
How many rows each rule touched. Counted independently — one row can break several rules at once.
Busiest zones
By published trips, attributed to the pickup zone.
The mean next to the median
Why the output carries both, plus a denominator: 122 rows out of 3.5 million hold 47 % of all miles driven, so the mean distance is overshot while the median does not move. Coverage says what share of the trips that mean is actually built from.
| Zone | Borough | Trips | Mean dist. | Median | Coverage | Mean fare |
|---|