← All Kits · All Guides

Reviewing a Cleaning Script Before It Touches Your Data

Michael Nocito · Updated August 2026 · Every number on this page was worked before it was published

A cleaning script is a list of decisions about what your data means, written as code. Drop these rows. Treat this blank as a zero. Assume these two spellings are the same company. Every one of those is defensible in some situation and wrong in others, and none of them raises an error.

What you do: for each operation, record what it changed and decide whether it is a fact or an assumption. Assumptions go in the write-up.

The short version. Print the counts before and after. The counts are the review.

The measurement harness

Before reviewing any step, add this around the script. Four lines, and they turn a silent transformation into an auditable one:

def snapshot(df, label):
    print(label, 'rows', len(df))
    print(df.isna().sum())
    print('distinct ids', df['customer_id'].nunique())

snapshot(raw, 'before')
clean = clean_it(raw)
snapshot(clean, 'after')

If the row count moved, something was dropped. If a null count went to zero, something was filled. If the distinct id count moved, identities were merged or split. Those three numbers describe most of what a cleaning script does.

The six operations, and what each one decides

1. dropna

df = df.dropna()          # every column, any null, row gone

The version with no arguments is the one to challenge. It drops a row if any column is null, including columns your analysis never uses. On a wide table that can remove most of the data. Check the count, then narrow it: dropna(subset=['amount']) at least states which column mattered.

The deeper question is what missing means. Missing at random is a nuisance. Missing because a field is only filled for one type of customer is a group being deleted, and the analysis afterwards is about a different population than the one you were asked about.

2. fillna

df['amount'] = df['amount'].fillna(0)

This creates data. A zero will be summed, averaged and plotted exactly like a measured zero, and nothing downstream can tell them apart. Fill with 0 only when missing means none. If missing means not recorded, leave it and let the aggregation skip it, or fill with a sentinel and record the count.

Median and mean fills carry the same warning with a friendlier face: they shrink variance and pull every group toward the middle, so any spread measure computed afterwards is understated.

3. replace and category mapping

df['region'] = df['region'].replace({'Nrth': 'North', 'N': 'North'})

Each mapping is a claim that two labels mean the same thing. Usually true, occasionally not: N might be North, or it might be New, or it might be a system default. Ask where the mapping came from. If the answer is that it was inferred from similarity, it needs checking against a real reference list. That is the subject of entity resolution.

Print the value counts before and after. A category that goes from 900 rows to 4,000 is either a good merge or a bad one, and the number is what starts the conversation.

4. Type casts

df['id'] = df['id'].astype(int)
df['date'] = pd.to_datetime(df['date'])

Casting an ID to a number destroys leading zeros permanently, and any ID longer than 15 digits loses precision. Date parsing without an explicit format guesses, and the classic failure is a day-first file read as month-first, which silently swaps 3 March and 1 April style values while raising nothing on the days above 12.

pd.to_datetime(df['date'], format='%d/%m/%Y', errors='raise')

errors='coerce' is worth a careful look wherever you see it: it turns unparseable dates into nulls, which the next dropna then deletes. Two innocent-looking lines, and the rows with the odd date format are gone.

5. drop_duplicates

df = df.drop_duplicates(subset=['customer_id'], keep='first')

Which one is first depends entirely on the sort order, and if the frame was not sorted deliberately, the row that survives is arbitrary. If the duplicates differ in any column, you have chosen a value at random. Sort explicitly by the column that decides, usually a timestamp, and say in the write-up that the latest record wins.

6. Clipping and outlier removal

df = df[df['amount'] < df['amount'].quantile(0.99)]

This removes the top one percent by construction, whether or not those rows are errors. On revenue data the top one percent is often the customers that matter most. Removing an outlier is only legitimate when you can say what makes it wrong, not merely that it is large. See percentiles and the outlier rule.

The review, as a table you fill in

Four rows of table, and the analysis now has a provenance section instead of a claim that the data was cleaned.

How to apply this to your own work

1. Add the snapshot function to every cleaning script you run, including short ones.
2. Never accept a bare dropna(). Ask which column mattered and how many rows it cost.
3. Challenge every fillna(0) with the question: does missing mean none here?
4. Give every date parse an explicit format, and treat errors='coerce' as something to justify.
5. Publish the before-and-after counts alongside the result. It is the cheapest credibility you will ever buy.

The one habit to keep

Keep the raw file. Cleaning should always be a script that runs from an untouched source, never an edit to the only copy. That single rule means every decision on this page is reversible, which is what makes reviewing them worth doing.

Could you re-run your last cleaning script from the original file today and get the same output?