Decision Trees
A decision tree predicts an outcome by asking a sequence of simple yes/no questions about the features, structured as a binary tree: each internal node tests a single feature against a threshold, each branch represents the outcome of that test, and each leaf node holds a prediction (a class label or a numeric value). Trees are built greedily and recursively — at each node, the algorithm searches over all features and thresholds for the split that best separates the data according to some impurity criterion, then repeats on each resulting subset. Their appeal is interpretability: a shallow decision tree can be read directly as a set of human-understandable rules, unlike the opaque coefficient vectors of linear models or the black-box nature of neural networks.
Cricket analogy: A decision tree predicting whether a batter will get out is like a captain's field-setting flowchart: 'Is the batter left-handed? Then is the ball short-pitched?' each yes/no question splits the data until a leaf gives a clear prediction, unlike an opaque analytics model.
Splitting Criteria: Gini Impurity and Entropy
For classification, the two dominant impurity measures are Gini impurity and entropy. Gini impurity for a node is 1 - sum(pi^2) over classes i, measuring the probability that a randomly chosen sample would be misclassified if labeled according to the node's class distribution; it is 0 for a pure node (all one class) and maximized when classes are evenly mixed. Entropy, borrowed from information theory, is -sum(pi * log2(pi)) and measures the disorder or unpredictability of the class distribution. Both criteria produce broadly similar trees in practice; a split is chosen to maximize the reduction in impurity (information gain) between the parent node and the weighted average impurity of its children. For regression trees, the criterion is typically variance reduction (mean squared error) rather than Gini or entropy.
Cricket analogy: Gini impurity is like measuring how mixed a fielding drill group is between left and right-handers — zero when everyone's the same type — while entropy measures the same disorder using an information-theory lens; a regression tree predicting final score instead splits to reduce variance in projected runs.
Controlling Tree Growth
An unconstrained decision tree will keep splitting until every leaf is perfectly pure (or contains a single sample), which almost always overfits severely — the tree memorizes noise in the training data rather than learning generalizable patterns. In practice, growth is controlled through hyperparameters: max_depth limits how many splits deep the tree can go, min_samples_split and min_samples_leaf require a minimum number of samples before a node can be split or form a leaf, and max_leaf_nodes caps the total number of leaves. Pruning (either pre-pruning via these constraints, or post-pruning by growing a full tree and then removing branches that don't improve validation performance) is essential to getting a decision tree that generalizes.
Cricket analogy: An unconstrained tree memorizing every quirky dismissal in the training data is like a coach who overreacts to one freak run-out and builds an entire strategy around it; max_depth caps how many questions deep the analysis goes, and pruning trims overreactions that don't hold up in the next series.
import numpy as np
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=500, n_features=5, n_informative=3, random_state=5)
feature_names = [f"feature_{i}" for i in range(5)]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=5)
# Unconstrained tree tends to overfit
full_tree = DecisionTreeClassifier(random_state=5)
full_tree.fit(X_train, y_train)
print("unconstrained depth:", full_tree.get_depth())
print("unconstrained train acc:", full_tree.score(X_train, y_train))
print("unconstrained test acc:", full_tree.score(X_test, y_test))
# Constrained tree with pruning-style hyperparameters
for depth in [2, 3, 4, 6, None]:
tree = DecisionTreeClassifier(max_depth=depth, min_samples_leaf=5, random_state=5)
scores = cross_val_score(tree, X_train, y_train, cv=5)
print(f"max_depth={depth} cv accuracy={scores.mean():.3f}")
best_tree = DecisionTreeClassifier(max_depth=4, min_samples_leaf=5, random_state=5)
best_tree.fit(X_train, y_train)
print(export_text(best_tree, feature_names=feature_names, max_depth=2))Decision trees naturally handle feature interactions and nonlinear relationships without any manual feature engineering, and they require no feature scaling at all, since splits are based on ordering thresholds rather than distances or magnitudes — a rare and convenient property among common algorithms.
Single decision trees are notoriously high-variance: a small change in the training data can produce a substantially different tree structure, especially near the root where early splits cascade into very different downstream partitions. This instability is precisely the motivation for ensemble methods like random forests, which average over many trees to stabilize predictions.
- Decision trees recursively split data using feature thresholds chosen to minimize impurity (Gini, entropy, or variance for regression).
- Leaf nodes hold final predictions; shallow trees are highly interpretable as human-readable rules.
- Unconstrained trees overfit by growing until nodes are pure, memorizing training noise.
- Hyperparameters like max_depth, min_samples_split, and min_samples_leaf control overfitting via pre-pruning.
- Trees require no feature scaling and naturally capture nonlinear relationships and interactions.
- Individual trees are high-variance and unstable, motivating ensemble methods like random forests.
Practice what you learned
1. What does Gini impurity of 0 at a node indicate?
2. What is the primary risk of allowing a decision tree to grow without any depth or leaf-size constraints?
3. Which hyperparameter directly limits how many levels deep a decision tree can grow?
4. Why do decision trees not require feature scaling, unlike k-NN or gradient-descent-based linear models?
5. What is the main motivation for ensemble methods like random forests, given properties of individual decision trees?
Was this page helpful?
You May Also Like
Random Forests and Ensembles
Explains how ensemble methods like random forests and boosting combine many weak or unstable models to produce more accurate, robust predictions.
k-Nearest Neighbors (k-NN)
Explains the instance-based k-NN algorithm, which classifies or predicts based on the majority vote or average of the k closest training examples.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.
Confusion Matrix and Classification Metrics
A structured breakdown of correct and incorrect predictions by class, forming the foundation for accuracy, precision, recall, and other classification metrics.