Feature Scaling and Normalization

Analogy categoryAnalogies

Feature scaling transforms numeric columns so they live on comparable numeric ranges before being fed into a model. Many algorithms measure distances or gradients directly on raw feature values, so a column like 'income' (ranging into the hundreds of thousands) can silently dominate a column like 'age' (ranging from 0 to 100) purely because of its larger magnitude, not because it is more predictive. Scaling removes this artifact so that a model's notion of 'important' reflects genuine signal rather than units of measurement.

🏏

Cricket analogy: A model comparing a bowler's economy rate (2-12) against career wickets (0-800) will let wickets silently dominate purely due to its larger numeric range, not because it's more predictive; scaling puts both stats on comparable footing so genuine signal, not units, drives the model.

Standardization vs. Min-Max Scaling

Standardization (z-score scaling) subtracts the mean and divides by the standard deviation, producing a feature with mean 0 and standard deviation 1: z = (x - mu) / sigma. It does not bound values to a fixed interval, which makes it forgiving of outliers relative to min-max scaling, and it is the default choice for algorithms that assume roughly Gaussian-shaped inputs, such as linear/logistic regression with regularization, PCA, and SVMs. Min-max scaling instead maps the minimum observed value to 0 and the maximum to 1 using x' = (x - min) / (max - min). This preserves the exact shape of the original distribution and is useful when you need bounded inputs, such as pixel intensities for a neural network, but a single extreme outlier can compress every other value into a tiny sliver of the [0, 1] range.

🏏

Cricket analogy: Standardizing a batter's average centers it at zero with unit spread, forgiving of one freak 250-run outlier innings, and suits regression-style rating models; min-max scaling instead squeezes strike rates into 0-1 exactly, useful for a bounded 'form meter' display but one huge outlier innings compresses everyone else into a tiny sliver.

Robust Scaling and When Scaling Is Unnecessary

Robust scaling centers on the median and scales by the interquartile range (IQR) rather than the mean and standard deviation, making it far less sensitive to extreme values than either standardization or min-max scaling. It is the right default when a dataset contains heavy-tailed features or known outliers you don't want to (or can't) remove. Not every model needs scaling at all: tree-based methods such as decision trees, random forests, and gradient-boosted trees split on threshold comparisons within a single feature at a time, so monotonic rescaling never changes which splits they choose. Applying scaling to tree models is harmless but adds no value.

🏏

Cricket analogy: Robust scaling centers a bowler's economy rate on the median and scales by the interquartile range, shrugging off one freak 15-runs-an-over outlier spell that would distort a mean-based measure; but a decision tree splitting on 'economy above 6' doesn't care about scaling at all since it only compares thresholds.

python

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
from sklearn.model_selection import train_test_split

X_train, X_test = train_test_split(np.random.default_rng(0).normal(50, 15, (200, 3)), test_size=0.2, random_state=42)

# Fit the scaler ONLY on training data, then apply to both splits
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)   # note: transform, not fit_transform

print('train mean ~0:', X_train_scaled.mean(axis=0).round(2))
print('train std ~1:', X_train_scaled.std(axis=0).round(2))

# Robust scaling for outlier-heavy features
robust = RobustScaler()
X_train_robust = robust.fit_transform(X_train)

Always fit the scaler on the training split only, then use that same fitted scaler to transform the validation and test splits. Fitting a scaler on the full dataset before splitting leaks statistics (the mean, std, min, max) from the test set into training, silently inflating your reported accuracy — a subtle but common form of data leakage.

Gradient descent-based optimizers converge much faster on scaled data because the loss surface becomes closer to a circular bowl rather than a stretched, narrow valley — with unscaled features, the optimizer zig-zags slowly toward the minimum along the elongated axis of the large-magnitude feature.

Scaling puts numeric features on comparable ranges so no feature dominates purely due to its units or magnitude.
Standardization (z-score) centers on mean 0 / std 1 and tolerates outliers better than min-max scaling.
Min-max scaling bounds values to [0, 1] but is sensitive to extreme outliers compressing the rest of the range.
Robust scaling uses median and IQR, making it the best choice for outlier-heavy data.
Distance-based and gradient-based models (KNN, SVM, linear/logistic regression, neural nets) need scaling; tree-based models generally don't.
Always fit the scaler on training data only, then transform validation/test data with that same fitted scaler to avoid data leakage.

Practice what you learned

1. Which scaling method is least sensitive to outliers?

2. Why must a scaler be fit only on the training set?

3. Which family of models is generally NOT affected by feature scaling?

4. What does min-max scaling compute?

5. Why does standardized data help gradient descent converge faster?

Was this page helpful?

Handling Missing Data

Train/Test Split and Cross-Validation

Topics covered

#Python#MachineLearningBasicsStudyNotes#MachineLearning#FeatureScalingAndNormalization#Feature#Scaling#Normalization#Standardization#StudyNotes#SkillVeris

Feature Scaling and Normalization