About
What this is, and what it is not
A static site that snapshots a handful of public datasets once a day and publishes only what changed between two snapshots.
The diff is the product
Each morning a GitHub Actions job fetches each dataset, normalizes it into records with stable keys, and compares it to the previous snapshot field by field. The result is stored three ways: the full normalized snapshot (overwritten), the day's diff (kept forever), and a status file per source. The site you are reading is rendered from that archive — no database, no server, no runtime dependencies.
Five rules the pipeline will not break
- “No snapshot” is not “no changes.” A failed fetch shows as a failed fetch, with the last good time, and produces no diff at all.
- A snapshot must earn acceptance. Record counts, required fields, key shapes and the share of records that moved are checked before anything is stored. A snapshot that fails is quarantined: yesterday's records stay, the reason is published, and no diff is emitted.
- Schema drift is announced. Each adapter declares the fields it expects. New or vanished fields produce a visible notice rather than silently rewriting the diff.
- Nothing is inferred. Every item shows the two snapshot times it came from. The site reports that a record changed; it never says why.
- Backfills are labelled. A record that appears today but carries an older event date is marked “backfilled”, not “new”.
The datasets
| Dataset | Publisher | License | Window | Link |
|---|---|---|---|---|
| Charlotte 311 service requests | City of Charlotte Open Data Portal | CC BY 4.0 | 14 days | source ↗ |
| Federal Register | Office of the Federal Register | Public domain (US Government work) | 21 days | source ↗ |
Privacy
311 requests and permit records can carry names and precise addresses. Personal fields are dropped or coarsened at normalization time, before anything reaches the repository. Which fields are kept, dropped or coarsened is written down per source in docs/privacy.md.
Being a good citizen
One request per source per day, a descriptive User-Agent with a contact address, conditional requests where the publisher supports them, and backoff that honors Retry-After. Terms, licenses and update cadence for each source are recorded in docs/sources.md.
Reading the terms used here
- appeared — in today's snapshot and not yesterday's.
- changed — present in both, with at least one comparable field different.
- disappeared — present yesterday, absent today and still inside the rolling window.
- aged out — left the rolling window because the window moved. Never counted as a deletion.
- backfilled — new to the snapshot, but dated before the previous snapshot was taken.
- quarantined — the snapshot failed a sanity gate and was not published at all.
Source and paper
Everything is on GitHub: github.com/slayyyyer/what-changed. The technical write-up is docs/paper.md; the decisions are in docs/adr/.
Built November 6, 2026 at 2:58 PM EST from commit dd50f61.