Handling Outliers

Analogy categoryAnalogies

Outliers are data points that deviate substantially from the bulk of a distribution — a $50,000 transaction in a dataset of mostly $20-200 purchases, a sensor spike from equipment malfunction, or simply a legitimate but rare extreme value. They matter because many summary statistics (mean, standard deviation, and models built on them like linear regression) are highly sensitive to extreme values, so a handful of outliers can distort an entire analysis. Handling outliers well requires first distinguishing genuine anomalies from valid extreme observations, since removing real data indiscriminately introduces its own bias.

🏏

Cricket analogy: A batter's freak 264-run knock in a dataset of mostly 20-40 run innings is a genuine outlier that can wildly distort a season's average, so it must be distinguished from a scoring error like a miskeyed 2640.

Detecting outliers with IQR and z-score

The interquartile range (IQR) method flags a value as an outlier if it falls below Q1 - 1.5*IQR or above Q3 + 1.5*IQR, where IQR = Q3 - Q1 and Q1/Q3 are the 25th/75th percentiles; it's robust to skew and doesn't assume normality, which makes it a common default. The z-score method standardizes each value as (x - mean) / std and flags values beyond a threshold (commonly |z| > 3); it's simple and interpretable but sensitive to the very outliers it's trying to detect, since extreme values inflate the mean and standard deviation used to compute it.

🏏

Cricket analogy: The IQR method flags a bowler's economy rate as an outlier if it falls far below Q1 or above Q3 of the team's rates, robust even if the distribution is skewed by a few very defensive spells, while the z-score method flags scores beyond 3 standard deviations from the mean, though that mean itself gets pulled by the outlier it's hunting.

python

import numpy as np
import pandas as pd

transactions = pd.DataFrame({
    'amount': [22.5, 19.0, 25.75, 21.0, 480.0, 18.5, 23.0, 20.25, 17.75, 26.0],
})

# --- IQR method ---
q1 = transactions['amount'].quantile(0.25)
q3 = transactions['amount'].quantile(0.75)
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr
iqr_outliers = transactions[(transactions['amount'] < lower) | (transactions['amount'] > upper)]
print(iqr_outliers)
#    amount
# 4   480.0

# --- Z-score method ---
mean, std = transactions['amount'].mean(), transactions['amount'].std()
z_scores = (transactions['amount'] - mean) / std
z_outliers = transactions[z_scores.abs() > 3]

# --- Capping (winsorizing) instead of removing ---
capped = transactions['amount'].clip(lower=lower, upper=upper)

Treatment strategies: remove, cap, or transform

Once flagged, outliers can be removed (dropping rows entirely, appropriate when they represent clear data errors like a negative age), capped/winsorized with clip(lower, upper) to pull extreme values in to a boundary rather than discarding them (preserving the row while limiting its influence), or handled by transforming the whole distribution — a log transform, for instance, compresses large values and often makes right-skewed data with legitimate extreme values look closer to normal without deleting any information.

🏏

Cricket analogy: A clearly impossible negative wicket count gets dropped outright, a freak 264-run innings gets capped with clip() to a reasonable ceiling to limit its influence on averages without erasing the record, and a log transform can tame a right-skewed run-scoring distribution overall.

The right treatment depends on the outlier's cause. Sensor glitches and obvious data-entry errors (negative prices, impossible dates) usually warrant removal. Genuine rare-but-real extreme values (a whale customer's huge order) are often better capped or log-transformed so the model still learns from them without being dominated by a handful of extreme rows.

Never apply outlier removal blindly to every numeric column with the same threshold. A z-score or IQR rule appropriate for a roughly normal distribution can flag a large fraction of a genuinely skewed distribution (like income or transaction size) as 'outliers,' silently discarding a meaningful chunk of real, informative data.

Outliers can distort means, standard deviations, and models trained on them, but not every outlier is an error worth removing.
The IQR method (Q1 - 1.5*IQR to Q3 + 1.5*IQR) is robust to skew and a common default for outlier detection.
The z-score method (|z| > 3 typically) is simple but itself sensitive to the outliers it's meant to detect, since they inflate the mean/std.
clip(lower, upper) caps (winsorizes) extreme values in place, preserving rows while limiting their influence.
Log or other monotonic transforms can reduce the impact of legitimate extreme values without discarding any data.
The correct treatment (remove, cap, transform, or leave alone) depends on the outlier's likely cause, not just a fixed statistical rule.

Practice what you learned

1. Under the standard IQR outlier rule, at what threshold is a value flagged as an outlier?

2. Why is the z-score method for outlier detection considered sensitive to the outliers it's trying to find?

3. What does Series.clip(lower=lo, upper=hi) do to values outside that range?

4. Which outlier treatment is generally most appropriate for a value that is clearly a data-entry error, such as a negative age?

5. Why might applying a uniform z-score or IQR outlier rule across many differently-shaped columns be problematic?

Was this page helpful?

String Methods in Pandas

Topics covered

#Python#PandasNumPyStudyNotes#DataScience#HandlingOutliers#Handling#Outliers#Detecting#IQR#StudyNotes#SkillVeris

Handling Outliers