Data Quality Is a Process, Not a Cleanup Project
Almost every organisation has run a data cleanup. Duplicates merged, formats normalised, missing values investigated, a satisfying report circulated. Within a quarter the same problems reappear, because the cleanup addressed the symptoms while the mechanism producing them kept running untouched.
Where defects enter
Rarely in the warehouse. Almost always at the boundaries: a free-text field where a controlled list belonged, an integration that changed its schema without notice, a manual export that someone edits before uploading. Engineering teams documenting these incidents, the data engineering notes here included, report the same distribution — most quality failures are upstream contract failures.
Controls that hold
- Automated tests on every load: row counts, null rates, uniqueness, referential integrity
- Schema contracts with producers, versioned, with breaking changes announced
- Freshness monitoring, since stale data looks correct and is not
- Quarantine for failing records instead of silent dropping
- A named owner per dataset who is paged when its tests fail
Fail loudly
The most damaging design choice in data engineering is the silent coercion — a malformed date becoming null, a mismatched join dropping a tenth of the rows, a default filling in for a missing value. Each keeps the pipeline green while corrupting the output. Loud failures cost an engineer an afternoon; quiet ones cost a quarter of decisions made on wrong numbers.
Making the case for the work
Quality work is hard to fund because success looks like nothing happening. Translate it into incidents avoided: the report that went to the board with a double-counted region, the campaign sent to churned customers, the forecast rebuilt twice. Attach hours and money to those, and the argument makes itself. Track the same figures afterwards, or the funding disappears the moment the incidents stop.