Handling Missing Values in pandas: dropna, fillna and What Each One Claims

1. Find them first, and count them

One line, and it is the only place the whole picture appears.

df.isna().sum()

order_id      0
order_date    0
rep           0
region        1
product       0
units         2
unit_price    1
revenue       3

128 cells in the frame, 7 of them missing

Seven cells out of a hundred and twenty-eight. Notice that revenue has three missing where units has two and unit_price has one: revenue is computed from both, so a hole in either produces a hole in the result. Missingness spreads through arithmetic, and a derived column is missing wherever any of its inputs is.

Two more lines are worth running at the same time. df.isna().mean() gives the proportion rather than the count, which is what you will quote. And df.isna().sum(axis=1) gives the count per row, which tells you whether the holes are concentrated in a few bad rows or scattered across many, and those two situations call for different treatment.

2. dropna, and what it takes with it

Before the explanation: seven cells are missing out of sixteen rows. Say how many rows survive dropna().

len(df)                          16
len(df.dropna())                 12
len(df.dropna(subset=["units"])) 14
len(df.dropna(how="all"))        16
df.dropna(axis=1).shape[1]        4 of 8 columns

Plain dropna() removed four rows, a quarter of the table, to deal with seven cells. Its default is how="any": a row goes if any column in it is missing, including columns you were not going to use. That is the most common way to throw away data by accident.

The three arguments that make it deliberate:

And axis=1 drops columns rather than rows, which on this frame leaves four of the eight standing. It is occasionally right for a wide import full of empty fields, and it is a very fast way to lose the column your analysis was about, so name the columns instead when you can.

Whichever you use, print the row count before and after. Losing a quarter of the table is a finding; losing it without noticing is the problem.

3. The summaries already skip them

Before you decide to fill or drop anything, know what pandas already does, because for a lot of work the answer is that you do not have to do either.

units, with two values missing
  count 14   sum 74.0   mean 5.2857   std 2.5246

the same column with nothing missing
  count 16   sum 101    mean 6.3125   std 3.7008

Every summary skipped the missing values rather than erroring or returning nothing. sum added the fourteen it had. mean divided by fourteen, not sixteen. And count reports fourteen, which is the point: count tells you how many values a summary was computed from, and comparing it with the row count is the cheapest missing-data check there is.

That behaviour is helpful and it is also the quiet part. A mean over a column that is one-eighth missing is a statement about the seven-eighths that answered, and it will be presented as a statement about everybody unless somebody says otherwise. Here the honest sentence is that average units are 5.29 among the fourteen orders where we know the quantity.

The knock-on for derived columns is worth seeing too. Revenue needed both units and price, so it exists on thirteen of the sixteen rows, and its total is 8,150 against a true 9,890. That 1,740 gap is not an error in the arithmetic; it is three rows that could not be computed.

4. Filling with zero

fillna(0) is the reflex, and it is a specific claim: that the missing quantity was none.

leave as NaN   count 14   sum 74.0   mean 5.2857   std 2.5246
fillna(0)      count 16   sum 74.0   mean 4.6250   std 2.9637

The sum did not move, because adding zero adds nothing. The mean fell from 5.29 to 4.63, because the denominator went from fourteen to sixteen. And the spread went up, from 2.52 to 2.96, because two values were placed at the bottom of the range.

So the rule is narrow: fill with zero only when zero is what happened. No sales that month, no returns that day, no clicks on that link. Those are real measurements that happen to be zero, and a database that stores them as missing is the thing that is wrong.

When the value is unknown rather than none, a zero is a false measurement that will be averaged, charted and summed exactly like a real one. This is the same trade as hiding an error behind a zero in a spreadsheet, and it goes wrong for the same reason.

5. Filling with the mean

Before the explanation: filling the two holes with the column's own mean cannot change the mean. Say what it does change.

leave as NaN     count 14   mean 5.2857   std 2.5246
fillna(mean)     count 16   mean 5.2857   std 2.3503

The mean is identical, to four decimal places, which is exactly why the method feels safe. The standard deviation fell from 2.52 to 2.35, and it had to: two values were placed precisely at the average, so they contribute nothing to the spread while still counting toward the divisor.

That is the picture at the top of the page. The filled points are not measurements, they are the average repeated, and the data now looks more consistent than it was ever observed to be. Anything downstream that reads the spread, an error bar, a confidence interval, a control limit, a correlation, is now reading a number that is too small, and nothing indicates it.

Mean filling has a place: a single missing value in a large, well-behaved column, where you need a complete column for a tool that will not accept holes, and where nobody will use the spread. Outside that, prefer to leave the hole, or to fill from something that actually knows: the same customer's other orders, the same product's usual quantity, the group median rather than the overall mean.

Say out loud which of your reports would change if the spread of one column were understated by seven percent. If none of them would, mean filling costs you nothing. If one of them is a control chart, it costs you the chart.

6. Forward fill, and the value it invents

ffill carries the previous value forward. On a genuine time series of something that persists, a price, a status, a meter reading, that is often exactly right: the value did not change, it simply was not re-recorded.

On anything else it invents data. Sorted by date, with one region missing:

order_date   region   region_ffill
2026-05-04   East     East
2026-05-11   NaN      East      <- the real answer was North
2026-05-18   South    South
2026-05-25   West     West

The filled value is confidently wrong. Nothing about the previous row's region tells you anything about this one, because region is a property of the order rather than a state that persists between orders.

Two guards make it safer where it does apply. limit=1 stops a single known value being carried across a long gap, which is what turns one missing reading into a week of flat line. And sorting explicitly before filling is not optional: ffill uses row order, so a frame that is not in time order fills from whatever happened to be above.

bfill is the same thing backwards and carries a stronger warning: it fills a value from the future, which in anything you will model or forecast is leakage.

7. The missing value that is not stored as missing

Before the explanation: a source system writes -1 where the quantity is unknown. Say what isna() reports.

Zero. It is a number, so nothing is missing as far as pandas is concerned.

with -1 as the marker   mean 4.5000   count 16   min -1
stored as missing       mean 5.2857   count 14

The mean is wrong by nearly one whole unit, the count says every row has a value, and the only visible clue is that the minimum is negative. Change the marker to 0 or 999 and even that clue disappears.

This is the worst case on the page precisely because none of the tools on the page apply: isna finds nothing, dropna drops nothing, and fillna has nothing to fill. The only defence is to look, once, at the values themselves.

df["units"].min(), df["units"].max()      absurd extremes
df["units"].value_counts().head()         one value appearing far too often
df.describe()                             a min or max that cannot be real

The habit that catches it: on any new numeric column, look at the minimum and the maximum, and ask whether both are possible. A quantity of −1, an age of 999, a date of 1900-01-01 and a price of 0.01 are all the same thing, and all of them are best converted to a real missing value at load time with na_values, which is the read_csv argument for exactly this.

8. groupby quietly drops the missing key

One row has no region. Grouping by region:

East   4
North  4
South  4
West   3
        rows accounted for: 15 of 16

Fifteen of sixteen. The row with no region is not a group, it is not an empty group, it is simply gone from the result, and no warning is printed. On a real table that can be a fifth of your revenue absent from every regional report with every printed number individually correct.

df.groupby("region", dropna=False)

East   4
North  4
South  4
West   3
NaN    1
        rows accounted for: 16 of 16

One argument, and the missing group appears. Run it that way once for every key you group by, read the NaN row, and then decide what to do about it rather than letting the default decide. That is the same habit and the same argument as in the groupby guide, and the check is arithmetic: group sizes must add back to the row count you started with.

Edge cases worth knowing

Six that come up.

NaN is not equal to itself. np.nan == np.nan is False, so df[df["region"] == np.nan] returns nothing while df["region"].isna().sum() returns one. Always use isna and notna, never an equality test.

An integer column becomes a float. Whole numbers plus a missing value cannot all be integers, so pandas widens the column and your quantities become 4.0 and 10.0. That is a signal rather than a fault, and the nullable integer type "Int64", with a capital I, keeps them whole if it matters.

Missing text is not an empty string. NaN and "" behave differently: one is skipped by counts and the other is counted. A column cleaned by filling text holes with "" will report more values than it has.

Filling in place, invisibly. A fillna early in a notebook affects everything after it, including cells you wrote before. Fill into a new column while exploring, so the original is still there to compare against.

Missing values in a merge key. They match each other, which multiplies rows. Remove or fill them before joining, not after.

Why it is missing matters. A quantity missing at random is a different problem from a quantity missing because the order was cancelled. The second one is information, and dropping those rows deletes a pattern rather than a nuisance.