Data Cleaning & Preprocessing Cheat Sheet
Practical pandas workflows for handling missing values, duplicates, outliers, and inconsistent data types before modeling.
2 PagesBeginnerMar 18, 2026
Missing Values & Duplicates
Detect, impute, and deduplicate rows.
python
import pandas as pddf = pd.read_csv("data.csv")# Inspect missingnessprint(df.isnull().sum())print(df.isnull().mean() * 100) # % missing per column# Drop rows/columns with too many missing valuesdf = df.dropna(thresh=len(df.columns) * 0.7) # keep rows with >=70% non-nulldf = df.drop(columns=["mostly_empty_col"])# Impute missing valuesdf["age"] = df["age"].fillna(df["age"].median())df["city"] = df["city"].fillna(df["city"].mode()[0])df["income"] = df.groupby("region")["income"].transform(lambda x: x.fillna(x.mean()))# Duplicatesprint(df.duplicated().sum())df = df.drop_duplicates(subset=["user_id"], keep="last")
Outliers & Data Types
Detect outliers and fix inconsistent columns.
python
# Detect outliers with the IQR methodQ1, Q3 = df["income"].quantile([0.25, 0.75])IQR = Q3 - Q1lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQRoutliers = df[(df["income"] < lower) | (df["income"] > upper)]# Cap (winsorize) instead of droppingdf["income"] = df["income"].clip(lower, upper)# Z-score methodz_scores = (df["income"] - df["income"].mean()) / df["income"].std()df = df[z_scores.abs() < 3]# Fix dtypes and inconsistent stringsdf["date"] = pd.to_datetime(df["date"], errors="coerce")df["price"] = pd.to_numeric(df["price"].str.replace("$", ""), errors="coerce")df["category"] = df["category"].str.strip().str.lower()
Cleaning Checklist
Standard steps to run on any new dataset.
- Missing values- check with isnull().sum(); impute (mean/median/mode) or drop based on missingness %
- Duplicates- detect with duplicated(), remove with drop_duplicates()
- Outliers- detect via IQR or z-score; decide to cap, transform, or remove based on domain knowledge
- Inconsistent types- coerce columns to the correct dtype with pd.to_numeric/pd.to_datetime
- Inconsistent categories- normalize casing/whitespace (e.g. 'NY' vs 'ny ' vs 'New York')
- Structural errors- fix typos, inconsistent units, or mislabeled columns
- Leakage columns- drop features that wouldn't be available at prediction time
- Class imbalance- check the target distribution before modeling; consider resampling or class weights
Imputation Strategies
How to fill in missing values responsibly.
- Mean/median imputation- simple and fast; median is more robust to skew and outliers
- Mode imputation- standard choice for categorical columns
- Group-wise imputation- fill using the mean/median within a related group (e.g. by region)
- Forward/backward fill- propagate the last/next known value; common for time series
- Model-based imputation- predict missing values from other features (e.g. KNNImputer, IterativeImputer)
- Missing indicator column- add a binary flag for 'was this value missing' to preserve that signal
Pro Tip
Never impute missing values or drop outliers before splitting into train/test sets — compute imputation statistics (median, mean) only on the training set, then apply them to test data, or you'll leak test-set information into training.
Was this cheat sheet helpful?
Explore Topics
#DataCleaningPreprocessing#DataCleaningPreprocessingCheatSheet#DataScience#Beginner#MissingValuesDuplicates#OutliersDataTypes#CleaningChecklist#ImputationStrategies#MachineLearning#CheatSheet#SkillVeris
