Handling Missing Data in Pandas

Analogy categoryAnalogies

Real-world data is rarely complete: sensors drop readings, survey respondents skip questions, and joins produce rows with no match on one side. pandas represents missing numeric and object data primarily with NumPy's float NaN (Not a Number), and increasingly with the nullable pd.NA for extension dtypes (Int64, boolean, string). Understanding how missingness propagates through comparisons and arithmetic, and knowing the standard toolkit for detecting and handling it, is essential before any aggregation, merge, or model can be trusted.

🏏

Cricket analogy: A rain-shortened match with no recorded overs for a bowler is like a missing NaN in the data; pandas marks that spot rather than guessing a fictitious figure so it doesn't quietly corrupt bowling averages.

Detecting missing values

isna() (alias isnull()) returns a boolean mask the same shape as the object, True wherever a value is missing; notna() is its complement. Calling .sum() on the mask gives a per-column count of missing values, a standard first step in any exploratory data analysis. Critically, NaN != NaN under IEEE 754 float comparison, so you must use isna() rather than == pd.NA or == float('nan') to detect missingness reliably.

🏏

Cricket analogy: isna() is like a scorer scanning the full scoresheet and circling every blank entry; summing those circles per bowler column tells you at a glance how many overs are missing before trusting any average.

python

import numpy as np
import pandas as pd

df = pd.DataFrame({
    'sensor_a': [21.4, np.nan, 22.1, 19.8],
    'sensor_b': [np.nan, np.nan, 30.5, 29.9],
})

print(df.isna().sum())
# sensor_a    1
# sensor_b    2
# dtype: int64

# Drop rows with ANY missing value
clean = df.dropna()

# Drop only rows missing in BOTH columns
clean_any_present = df.dropna(how='all')

# Fill with column mean
filled = df.fillna(df.mean())

# Forward-fill along time, then interpolate remaining gaps linearly
interp = df.interpolate(method='linear', limit_direction='forward')

Dropping vs. filling vs. interpolating

dropna() removes rows (or columns, with axis=1) containing missing values, controlled by how ('any' or 'all') and thresh (minimum number of non-null values required to keep the row). fillna() replaces missing values with a constant, a per-column dictionary of values, or a computed statistic like the mean or median; method='ffill'/'bfill' propagates the last valid observation forward or backward, common in time series. interpolate() estimates missing values based on surrounding data using linear, polynomial, or time-aware methods, often more defensible than a flat mean fill for ordered or time-series data.

🏏

Cricket analogy: dropna() might discard a player's row entirely if too many innings are missing, while fillna(method='ffill') carries forward a batter's last known strike rate, and interpolate() estimates a missing over's run rate from the overs around it.

There's no universally correct way to handle missing data — the right choice depends on why the data is missing. Data missing completely at random can often be safely dropped or mean-filled; data missing systematically (e.g. a sensor that fails at high temperatures) requires domain knowledge, since naive imputation can quietly bias downstream analysis.

fillna(0) is a common but dangerous default: zero is rarely a neutral placeholder for a real-valued measurement (like temperature or revenue) and can silently distort means, sums, and models. Prefer an explicit, justified strategy — a computed statistic, forward-fill, interpolation, or a flag column marking imputed values — over a reflexive fillna(0).

pandas represents missing data primarily with NaN (float) and increasingly pd.NA for nullable extension dtypes.
isna()/notna() are the reliable way to detect missing values; equality comparisons against NaN do not work.
dropna() removes rows/columns with missing values, tunable via how='any'/'all' and thresh.
fillna() replaces missing values with constants, statistics, dict-based per-column values, or forward/backward fill.
interpolate() estimates gaps from surrounding data, often more appropriate than a flat fill for ordered/time-series data.
The right missing-data strategy depends on the mechanism of missingness, not just convenience — fillna(0) is frequently the wrong default.

Practice what you learned

1. Why can't you reliably detect missing values in pandas using `df['col'] == np.nan`?

2. What does df.dropna(thresh=3) do?

3. Which fillna approach is generally most defensible for a gap in an ordered time-series measurement?

4. What does fillna(method='ffill') do?

5. Why is fillna(0) often considered risky for numeric measurement columns?

Was this page helpful?

Data Type Conversion

Topics covered

#Python#PandasNumPyStudyNotes#DataScience#HandlingMissingDataInPandas#Handling#Missing#Data#Pandas#StudyNotes#SkillVeris

Handling Missing Data in Pandas