Common Machine Learning Pitfalls
Most machine learning failures aren't caused by picking the 'wrong' algorithm — they're caused by process mistakes that silently corrupt an otherwise reasonable pipeline: information leaking from the future or from the test set, evaluating with a metric that doesn't match the business problem, or drawing conclusions from a model that was never actually tested on realistic data. These pitfalls are dangerous precisely because they often still produce a working model with a plausible-looking accuracy number — the failure is invisible until the model underperforms in production or an auditor asks the right question.
Cricket analogy: Most series losses aren't from picking the wrong playing XI but from process failures like a practice pitch that doesn't match match conditions; the team can still post good net-session numbers, and the failure stays invisible until it collapses on the real pitch under pressure.
Data Leakage
Data leakage happens when information that wouldn't be available at prediction time sneaks into training, making a model look far better during evaluation than it will in production. Classic examples: fitting a scaler or imputer on the full dataset before splitting into train/test, including a feature that is itself derived from the target (e.g., 'days until cancellation' when predicting churn), or performing feature selection using the entire dataset before cross-validation. The fix is procedural discipline — split the data first, and ensure every preprocessing step is fit exclusively on the training fold, ideally enforced structurally with a Pipeline.
Cricket analogy: Data leakage is like setting a form benchmark using stats from the entire season including matches not yet played, or a player-of-the-match predictor literally derived from the final scorecard; the fix is procedural discipline — lock in selection before the match, using only information available at toss time.
Choosing the Wrong Metric
Optimizing and reporting accuracy on an imbalanced dataset is one of the most common and most misleading mistakes in applied ML. A model that predicts the majority class 100% of the time can score 99% accuracy on a dataset where the minority class occurs 1% of the time, while being completely useless. The deeper pitfall is not knowing the metric is wrong until a stakeholder points out the model never actually catches the rare, important cases. Metric choice should be driven by the actual cost of different error types, not by which number happens to look most impressive.
Cricket analogy: Optimizing prediction accuracy for a rain-affected tournament by always predicting no-result scores impressively high when abandonments are common, but is useless the moment a real result matters; metric choice should reflect what actually costs the broadcaster money, not which number looks best in a press release.
Overfitting to the Validation Set
It's possible to overfit not just to training data, but to a validation set, by repeatedly tuning hyperparameters against the same validation split until performance on it is inflated by chance. This is why a truly held-out test set — touched only once, at the very end — is essential; validation performance during iterative tuning should be treated as an estimate, not a guarantee, of final generalization. Kaggle competitions illustrate this vividly through the gap that regularly appears between public and private leaderboard scores.
Cricket analogy: Repeatedly adjusting a batting order based on the same three warm-up matches until it looks perfect is like overfitting to a validation set; a real, untouched tour often exposes a gap between how the lineup performed in familiar warm-ups versus the actual series.
Ignoring Distribution Shift
A model evaluated on data drawn from the same distribution and time period as training data can still fail in production if the real-world data distribution shifts afterward — new customer segments, seasonal effects, or changed upstream data pipelines. Time-series and production ML systems are especially vulnerable to this, and a common related mistake is randomly shuffling time-ordered data into train/test splits, which lets the model 'see the future' during training and produces artificially strong offline results.
Cricket analogy: A batting model trained entirely on flat, dry subcontinental pitches can fail once conditions shift to a seaming English pitch mid-season; a related mistake is randomly shuffling a player's career innings into train/test sets instead of chronological order, letting the model see later peak form while evaluating an earlier period.
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
import numpy as np
X = np.random.randn(1000, 10)
y = (X[:, 0] + X[:, 1] > 0).astype(int)
# WRONG: fit scaler on all data before splitting -> leakage
# scaler = StandardScaler().fit(X)
# X_scaled = scaler.transform(X)
# X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.2)
# RIGHT: split first, fit preprocessing only on the training fold
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
pipe = Pipeline([
('scaler', StandardScaler()), # fit only on X_train internally
('clf', LogisticRegression()),
])
pipe.fit(X_train, y_train)
print('honest test accuracy:', pipe.score(X_test, y_test))A useful gut-check before trusting any reported model performance: ask 'would every piece of information used to make this prediction actually be available, in that exact form, at the moment we'd need to make the prediction in production?' If the answer is no for even one feature, leakage is likely present.
Beware of 'accidentally great' results. If a model's validation accuracy is dramatically higher than you expected given the problem's difficulty, the most likely explanation is a leakage or evaluation bug — not a breakthrough model. Investigate suspiciously good numbers before celebrating them.
- Data leakage — information from the test set or the future leaking into training — is the most common source of falsely optimistic results.
- Preprocessing steps must be fit only on training data, ideally enforced with a Pipeline.
- Accuracy is misleading on imbalanced datasets; choose metrics that reflect the real cost of different error types.
- Repeatedly tuning against the same validation set can overfit to that set; a final held-out test set should be touched only once.
- Randomly shuffling time-ordered data into train/test splits lets models 'see the future,' inflating offline performance.
- Suspiciously high performance is a signal to investigate for bugs, not a reason to celebrate immediately.
Practice what you learned
1. Which of the following is a classic example of data leakage?
2. Why is accuracy often a misleading metric on a highly imbalanced dataset?
3. What problem arises from repeatedly tuning hyperparameters against the same validation set?
4. Why is randomly shuffling time-ordered data before a train/test split risky?
5. What is a reasonable reaction to a model achieving suspiciously high validation performance on a historically difficult problem?
Was this page helpful?
You May Also Like
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.
Train/Test Split and Cross-Validation
Understand how holding out data and using k-fold cross-validation give an honest estimate of how a model will perform on unseen data.
Confusion Matrix and Classification Metrics
A structured breakdown of correct and incorrect predictions by class, forming the foundation for accuracy, precision, recall, and other classification metrics.
Cross-Validation Strategies
Learn how k-fold, stratified, and time-series cross-validation give more reliable estimates of model performance than a single train-test split.