Gradient Descent Cheat Sheet

The mechanics of gradient descent optimization, covering batch, stochastic, and mini-batch variants plus momentum and adaptive learning rate methods.

2 PagesIntermediateMar 8, 2026

Gradient Descent From Scratch

Minimize MSE loss for linear regression.

python

import numpy as npdef gradient_descent(X, y, lr=0.01, epochs=1000):    n, m = X.shape    weights = np.zeros(m)    bias = 0.0    for epoch in range(epochs):        y_pred = X @ weights + bias        error = y_pred - y        # Gradients of MSE loss w.r.t. weights and bias        grad_w = (2 / n) * X.T @ error        grad_b = (2 / n) * np.sum(error)        # Update parameters in the direction that reduces loss        weights -= lr * grad_w        bias -= lr * grad_b        if epoch % 100 == 0:            loss = np.mean(error ** 2)            print(f"Epoch {epoch}: loss={loss:.4f}")    return weights, bias

PyTorch Optimizers

SGD with momentum and Adam in a training loop.

python

import torch.nn as nnimport torch.optim as optimmodel = nn.Linear(10, 1)criterion = nn.MSELoss()# SGD with momentum: smooths updates using a moving average of past gradientsoptimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)# Adam: adapts the learning rate per parameter using 1st/2nd moment estimatesoptimizer = optim.Adam(model.parameters(), lr=0.001, betas=(0.9, 0.999))for epoch in range(100):    optimizer.zero_grad()          # clear old gradients    output = model(X_batch)    loss = criterion(output, y_batch)    loss.backward()                # compute gradients via backprop    optimizer.step()               # update weights

Gradient Descent Concepts

Variants and core vocabulary of the algorithm.

Batch gradient descent- computes the gradient over the entire dataset per update; stable but slow for large data
Stochastic gradient descent (SGD)- updates using one sample at a time; noisy but fast and can escape local minima
Mini-batch gradient descent- updates using small batches (e.g. 32-256); standard in deep learning, balances speed and stability
Learning rate- step size for each update; too high diverges, too low converges slowly
Momentum- accumulates a moving average of past gradients to smooth updates and speed convergence
Adam- combines momentum with per-parameter adaptive learning rates; a common default optimizer
Learning rate schedule/decay- reduces the learning rate over training to fine-tune convergence
Vanishing/exploding gradients- gradients shrink or grow uncontrollably in deep networks, hindering training

Optimizer Comparison

Trade-offs between common optimizers.

SGD- simple, generalizes well, but sensitive to learning rate and can be slow to converge
SGD + Momentum- accelerates convergence and dampens oscillations in ravines
RMSprop- adapts learning rate per parameter using a moving average of squared gradients; good for RNNs
Adam- combines momentum and RMSprop-style adaptive rates; fast convergence, a common default choice
AdamW- Adam with decoupled weight decay; often preferred for transformer training

Pro Tip

If training loss oscillates wildly or diverges (NaN), your learning rate is almost always too high — halve it before assuming there's a bug in your model architecture, and consider gradient clipping for RNNs/transformers where exploding gradients are common.

Was this cheat sheet helpful?

Ensemble Methods Cheat Sheet

Regularization (L1/L2) Cheat Sheet

Explore Topics

#GradientDescent#GradientDescentCheatSheet#DataScience#Intermediate#GradientDescentFromScratch#PyTorchOptimizers#GradientDescentConcepts#OptimizerComparison#Functions#MachineLearning#CheatSheet#SkillVeris

Sri Hayavadhana Info-Tech

Professional Web Designing Services