Deep Learning Optimizers Cheat Sheet

Compares SGD, Momentum, RMSprop, Adam, and AdamW update rules, and shows how to configure each optimizer in PyTorch with typical hyperparameters.

2 PagesIntermediateMar 12, 2026

Optimizer Families

How each optimizer adapts the learning rate or update direction.

SGD- Updates weights using the gradient of a mini-batch: w -= lr * grad; simple but sensitive to learning rate choice
SGD with Momentum- Accumulates a velocity term from past gradients to smooth updates and speed convergence through ravines
RMSprop- Divides the learning rate by a moving average of recent squared gradients, adapting per-parameter step size
Adam- Combines momentum (first moment) and RMSprop-style scaling (second moment) with bias correction
AdamW- Adam with decoupled weight decay, applied directly to weights instead of folded into the gradient -- the modern default for transformers
Learning rate schedule- Adjusts the learning rate over training (step decay, cosine annealing, warmup) independent of the optimizer's own adaptation

Configuring Optimizers in PyTorch

Typical setup and hyperparameters for common optimizers.

python

import torch.optim as optim# Vanilla SGD with momentumopt = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=1e-4)# Adam - good default for most non-transformer modelsopt = optim.Adam(model.parameters(), lr=1e-3, betas=(0.9, 0.999), eps=1e-8)# AdamW - standard choice for training transformersopt = optim.AdamW(model.parameters(), lr=5e-4, weight_decay=0.01)# Cosine annealing schedule on top of any optimizerscheduler = optim.lr_scheduler.CosineAnnealingLR(opt, T_max=50)

Adam Update Rule

The core math Adam performs each step (bias-corrected first/second moments).

python

# m, v initialized to zero; t = timestep; g = gradientm = beta1 * m + (1 - beta1) * g          # 1st moment (mean)v = beta2 * v + (1 - beta2) * (g ** 2)   # 2nd moment (uncentered variance)m_hat = m / (1 - beta1 ** t)             # bias correctionv_hat = v / (1 - beta2 ** t)w -= lr * m_hat / (v_hat ** 0.5 + eps)

Choosing an Optimizer

Rules of thumb from practice.

Default starting point- Adam or AdamW with lr around 1e-3 (CNNs/MLPs) or 1e-4 to 5e-4 (transformers)
When SGD+momentum wins- Often generalizes better than Adam on large-scale image classification given enough tuning and a good LR schedule
Gradient clipping- Clip gradient norm (e.g., to 1.0) to stabilize RNN/transformer training regardless of optimizer
Warmup- Linearly ramp the learning rate up for the first few hundred/thousand steps before decaying -- critical for AdamW on transformers

Pro Tip

AdamW's weight decay is decoupled from the gradient update -- if you switch from Adam to AdamW, re-tune weight_decay separately rather than reusing the same value, since it now behaves like true L2 regularization.

Was this cheat sheet helpful?

Monte Carlo Simulation Cheat Sheet

Batch Normalization Cheat Sheet

Explore Topics

#DeepLearningOptimizers#DeepLearningOptimizersCheatSheet#DataScience#Intermediate#OptimizerFamilies#ConfiguringOptimizersInPyTorch#AdamUpdateRule#ChoosingAnOptimizer#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services