Decision Tree
A decision tree is a supervised learning model that predicts an outcome by learning a hierarchy of simple, interpretable if-then decision rules based on input features, structured as a tree of splits leading to leaf predictions.
Definition
A decision tree is a supervised learning model that predicts an outcome by learning a hierarchy of simple, interpretable if-then decision rules based on input features, structured as a tree of splits leading to leaf predictions.
Overview
A decision tree is built by recursively splitting the training data into increasingly pure subsets, where each internal node tests a condition on a single feature (for example, 'is age greater than 30?'), and each leaf node holds a prediction — a class label for classification or a numeric value for regression. The tree is grown by choosing, at each step, the feature and threshold that best separates the data according to a splitting criterion: Gini impurity or information gain (entropy reduction) for classification, and variance reduction for regression. Grown without constraint, a decision tree can keep splitting until every leaf contains a single training example, which perfectly fits the training data but generalizes poorly — a classic overfitting failure mode. To control this, trees are typically regularized through pruning (removing branches that don't improve validation performance) or by limiting hyperparameters such as maximum depth, minimum samples per leaf, or minimum samples required to split a node. Decision trees are prized above almost any other model type for interpretability: the path from root to leaf for any prediction can be read directly as a sequence of human-understandable rules, which matters in regulated domains like credit lending or healthcare where model decisions must be explainable. They require little data preprocessing — no feature scaling is needed, and they naturally handle both numerical and categorical features. However, a single decision tree is a relatively weak, high-variance model on its own, since small changes in training data can produce very different tree structures. This instability is precisely what ensemble methods exploit: Random Forest averages many decorrelated trees to reduce variance, while Gradient Boosting chains trees sequentially to reduce bias, and in practice decision trees are used far more often as building blocks within these ensembles than as standalone models.
Key Concepts
- Learns hierarchical if-then splitting rules from training data
- Splitting criteria include Gini impurity, information gain, and variance reduction
- Highly interpretable — predictions can be traced as a path of human-readable rules
- No feature scaling required; handles numerical and categorical features natively
- Prone to overfitting without regularization (pruning, max depth, min samples per leaf)
- High variance — small data changes can produce very different tree structures
- Serves as the base learner for ensemble methods like Random Forest and Gradient Boosting
- Supports both classification (leaf = class label) and regression (leaf = numeric value)