Hyperparameter Tuning
Hyperparameters are the configuration choices set before training begins, such as a model's learning rate, regularization strength, tree depth, or number of neighbors, as opposed to parameters like weights or coefficients that are learned directly from the training data. Because hyperparameters strongly influence a model's ability to fit the data and generalize to new data, choosing them well is often the difference between a mediocre model and a state-of-the-art one, and this choice must be made using data the model has not been fit on, typically via cross-validation.
Cricket analogy: A captain decides field placements and bowling order (hyperparameters) before the match starts, while a batsman's actual shot timing develops during play (parameters learned from data); getting the pre-match plan right, tested against past matches, often separates a win from a loss.
Grid Search vs Random Search
Grid search exhaustively evaluates every combination of hyperparameter values from a predefined discrete set, guaranteeing that the best combination within the grid is found, but its cost grows exponentially with the number of hyperparameters and their possible values. Random search instead samples a fixed number of hyperparameter combinations from specified distributions; despite its apparent crudeness, research has shown it typically finds comparably good or better configurations than grid search with far fewer evaluations, especially when only a few hyperparameters actually matter for performance, since random search explores a wider range of values for each one.
Cricket analogy: Testing every single combination of batting order and bowling change across a full grid of options guarantees the best plan but takes forever to trial; randomly sampling a handful of promising combinations, as many modern analytics teams now do, often finds a nearly-as-good plan far faster.
Bayesian and Other Smarter Search Strategies
Bayesian optimization builds a probabilistic surrogate model (commonly a Gaussian process) of the relationship between hyperparameters and validation performance, using it to intelligently choose which combination to try next by balancing exploration of uncertain regions against exploitation of promising ones. This typically finds strong configurations with far fewer trials than grid or random search, which is valuable when each training run is expensive. Successive halving and Hyperband further speed up tuning by allocating more resources (e.g. training epochs) to promising configurations and eliminating poor ones early.
Cricket analogy: Rather than trialing every possible batting order blindly, a data-savvy coach builds a model predicting which orders tend to score well, using it to pick the next order to actually try — like Bayesian optimization; meanwhile giving only a few overs to weak trial lineups before cutting them, like Hyperband, saves practice time.
from sklearn.model_selection import RandomizedSearchCV
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import randint
param_distributions = {
'n_estimators': randint(50, 500),
'max_depth': randint(3, 30),
'min_samples_split': randint(2, 20),
'max_features': ['sqrt', 'log2', None],
}
search = RandomizedSearchCV(
estimator=RandomForestClassifier(random_state=42),
param_distributions=param_distributions,
n_iter=50,
cv=5,
scoring='f1',
n_jobs=-1,
random_state=42,
)
search.fit(X_train, y_train)
print("Best params:", search.best_params_)
print("Best CV score:", search.best_score_)
best_model = search.best_estimator_A useful mental model: parameters are what the model learns (e.g. regression coefficients), hyperparameters are what you, the practitioner, set before training (e.g. learning rate, number of trees) — tuning is the process of searching over hyperparameter choices to find the ones that generalize best.
Never tune hyperparameters using performance on the test set — doing so leaks test information into model selection and produces an overly optimistic estimate of real-world performance. Always tune using a separate validation set or cross-validation, and reserve the test set purely for final, one-time evaluation.
- Hyperparameters are set before training (e.g. learning rate, tree depth, k), unlike parameters, which are learned from data.
- Grid search exhaustively tries every combination; it is thorough but scales poorly with more hyperparameters.
- Random search samples combinations and often matches or beats grid search with far fewer evaluations.
- Bayesian optimization uses a probabilistic model to intelligently pick the next combination to try, reducing wasted trials.
- Hyperparameter tuning must use validation data or cross-validation, never the final test set.
- Techniques like successive halving/Hyperband save compute by cutting off poor configurations early.
Practice what you learned
1. What is the key difference between a model's parameters and its hyperparameters?
2. Why does random search often perform comparably to or better than grid search in practice?
3. What data should be used to select the best hyperparameter configuration?
4. What advantage does Bayesian optimization offer over grid or random search?
5. In scikit-learn, which class performs randomized sampling of a hyperparameter distribution combined with cross-validation?
Was this page helpful?
You May Also Like
Cross-Validation Strategies
Learn how k-fold, stratified, and time-series cross-validation give more reliable estimates of model performance than a single train-test split.
Gradient Descent Explained
Understand how gradient descent iteratively adjusts model parameters to minimize a loss function, and how learning rate and variants like SGD affect convergence.
Bias-Variance Tradeoff
A foundational concept explaining how model error decomposes into bias, variance, and irreducible noise, and how balancing them guides model complexity choices.
Overfitting and Underfitting
The two failure modes of model fitting — memorizing noise in training data versus failing to capture real patterns — and the diagnostic and mitigation techniques for each.