← All Kits · Exploratory Data Analysis · SQL Kit

Your Data Is Never Fully Clean — and Professionals Ship Anyway

Limitations sections, quality logs, and quantified caveats · Part of the Analyst Prep Kit

Here is a truth that surprises every beginner: data cleaning never finishes. You fix the import bug, standardize the messy names, handle the missing values — and there are still duplicates you chose not to merge, edge cases your rules miss, and quirks you haven't found yet. So how do professionals ever ship an analysis? Not by reaching perfection. By documenting every imperfection and making it defensible. This guide explains how that works, at the three levels real analysts use, with examples from open projects you can read.

The standard is fitness, not purity

The data-quality research field settled this decades ago: quality is not an absolute property of data, it's fitness for use — whether the data is good enough for the specific decision it feeds (Wang & Strong, 1996). The same customer table can be excellent for counting customers and useless for mailing them, if the addresses are stale. That reframing changes the analyst's job. The question is never "is this data clean?" (answer: no, nothing is). The question is "is it clean enough for THIS conclusion — and have I told the reader where it isn't?"

A concrete picture of why perfection is unreachable, from a real project: an analysis of Billboard chart history had to standardize artist names, because one artist can appear under seven different credit spellings. The cleaning rules fixed most of it — and deliberately left some of it. Duo names containing "&" were never split (splitting would shred real bands like Hall & Oates into fake solo artists), which means a duo member's work never merges with their solo career. That's not a bug someone missed. It's a documented tradeoff: the alternative rule caused worse damage, the cost was measured and written down, and the analysis proceeds with eyes open. Multiply that by every rule in every project and you see the professional reality — a finished analysis is a stack of defensible tradeoffs, not an absence of flaws.

Level 1: the limitations section (in the deliverable)

Every serious analytical deliverable — report, README, dashboard notes, academic paper — carries a section stating what the analysis can NOT claim and why. Common names: Limitations, Scope & Assumptions, Caveats. Academic journals require one in every paper; industry analysts who skip it get their conclusions stretched past what the data supports, and then own the wreckage.

You can read real ones in the wild: the Steam Hidden Gems project and the Streaming Hidden Gems project each carry a "Scope & Assumptions" section in their READMEs, stating things like which records were excluded and why, what the source data cannot see, and which cleaning decisions carry known costs.

What belongs in it:

• Deliberate tradeoffs — rules you chose knowing they have a cost (the "&" decision above).
• Known unfixables — problems you found and could not resolve (aliases, gaps in the source's coverage).
• Boundary conditions — what the data structurally can't answer ("this dataset ends in June; nothing here speaks to later months").
• Population limits — who or what the data doesn't represent.

Level 2: the data-quality log (the working record)

The limitations section is the public summary. Behind it, professionals keep a running log of every issue found while working: what the clue was, how it was confirmed, what was decided. Months later, when someone asks "why does artist X appear twice?", the answer is in the log instead of in a departed analyst's memory. In regulated industries this isn't optional — data lineage and decision audit trails are compliance requirements.

A useful format, per issue: the clue (what looked off), the validation (the query or check that confirmed it), the decision (what was done, or deliberately not done), and the concept (what class of problem this was, so you recognize it next time). An issue log written this way doubles as a learning document — each entry is a small case study in data forensics.

Level 3: quantified caveats (limitations with numbers)

The strongest limitation statements carry a measurement. Compare:

The pattern: when you find a limitation, spend one more query measuring its size. A limitation with a number is a deliverable in itself — it tells the reader exactly how much weight the conclusion can bear. A limitation without one forces the reader to guess, and readers guess pessimistically.

The trust paradox: admitting flaws makes you more credible

Beginners hide limitations, fearing they make the work look weak. Experienced reviewers read it exactly backwards. An analysis that claims no limitations wasn't examined closely; an analysis that says "here are the six problems I found, how I confirmed each one, what each cost, and why the conclusion survives them" was. A reader who finds a flaw you already documented trusts you more. A reader who finds a flaw you didn't mention stops trusting everything else.

This is also an interview weapon. "Walk me through your project" answered with findings alone is a book report. Answered with findings plus "and here's what the data couldn't tell me, and how I bounded that" — that's the answer that sounds like a working analyst, because accounting for uncertainty is most of what working analysts do.

How to write yours: a starter template

## Limitations & Assumptions

1. [Deliberate tradeoff] We did X instead of Y because Y caused
   worse damage (evidence: ...). Cost of X: ... (measured: N rows /
   M% affected).
2. [Known unfixable] The source data cannot distinguish A from B.
   Conclusions about A alone are therefore approximate.
3. [Boundary] Data covers [range]. Nothing here speaks to [outside].
4. [Match/coverage rate] Step Z joined/matched N% of records. The
   unmatched remainder looks like: ...

Write it as you work, not at the end — each entry takes one minute when the issue is fresh and an afternoon of archaeology when it isn't. The habit pairs with exploratory data analysis: exploration is how you find the issues; the limitations ledger is how you account for them. Finding without accounting wastes the find.

References

1. Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33. doi:10.1080/07421222.1996.11518099
2. Tukey, J. W. (1977). Exploratory Data Analysis. Addison-Wesley. (The exploratory tradition this guide's companion piece covers.)