Gradient Boosting Cheat Sheet

A reference for gradient boosting covering XGBoost, LightGBM, and scikit-learn implementations, plus learning rate tuning, early stopping, and regularization.

2 PagesIntermediateFeb 28, 2026

XGBoost

Train with early stopping on a validation set.

python

import xgboost as xgbmodel = xgb.XGBClassifier(    n_estimators=500, learning_rate=0.05, max_depth=4,    subsample=0.8, colsample_bytree=0.8,    eval_metric='logloss', early_stopping_rounds=20)model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)

LightGBM

Train using LightGBM's native Dataset API.

python

import lightgbm as lgbtrain_data = lgb.Dataset(X_train, label=y_train)val_data = lgb.Dataset(X_val, label=y_val, reference=train_data)params = {'objective': 'binary', 'metric': 'auc', 'num_leaves': 31, 'learning_rate': 0.05}model = lgb.train(    params, train_data, num_boost_round=1000,    valid_sets=[val_data],    callbacks=[lgb.early_stopping(stopping_rounds=50)])

scikit-learn GradientBoosting

Built-in gradient boosting without extra dependencies.

python

from sklearn.ensemble import GradientBoostingClassifiergb = GradientBoostingClassifier(    n_estimators=200, learning_rate=0.1, max_depth=3, subsample=0.9)gb.fit(X_train, y_train)

Key Concepts

Core theory behind gradient boosting.

Boosting- Sequentially fits trees to correct the residual errors of previous trees, unlike bagging's parallel trees
Learning rate (shrinkage)- Scales each tree's contribution; lower values need more trees but usually generalize better
Early stopping- Halts training once a validation metric stops improving, preventing overfitting
Regularization (XGBoost)- gamma, reg_alpha (L1), and reg_lambda (L2) penalize tree complexity directly in the loss function
Row/column subsampling- subsample and colsample_bytree add bagging-style randomness to each boosting round

Pro Tip

Tune learning_rate and n_estimators together — halve the learning rate and roughly double n_estimators, using early stopping on a held-out validation set, to trade extra compute for better generalization.

Was this cheat sheet helpful?

Random Forest Cheat Sheet

Support Vector Machines Cheat Sheet

Explore Topics

#GradientBoosting#GradientBoostingCheatSheet#DataScience#Intermediate#XGBoost#LightGBM#ScikitLearnGradientBoosting#KeyConcepts#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services