Gradient Descent Explained

Analogy categoryAnalogies

Gradient descent is the optimization algorithm that powers the training of most machine learning models, from linear regression to deep neural networks. The core idea is simple: given a loss function that measures how wrong the model's predictions are, compute the gradient (the vector of partial derivatives) of that loss with respect to each parameter, then nudge the parameters a small step in the direction that decreases the loss fastest — the negative gradient direction. Repeating this process iteratively moves the parameters toward a minimum of the loss surface.

🏏

Cricket analogy: A batsman adjusts his stance after each ball based on how far off it felt from ideal timing, nudging his technique a little each time in the direction that reduces mistimed shots, over many balls converging on a solid technique.

The Update Rule

At each iteration, parameters are updated as theta = theta - alpha * gradient(loss, theta), where alpha is the learning rate, a hyperparameter controlling the step size. If alpha is too small, training converges very slowly, wasting compute and time. If alpha is too large, updates can overshoot the minimum and cause the loss to oscillate or diverge entirely. Many modern optimizers, such as Adam and RMSprop, adapt the effective learning rate per parameter based on the history of gradients, making them more robust to a poorly chosen initial alpha.

🏏

Cricket analogy: A batsman making tiny stance tweaks each net session takes forever to fix a flaw, while overcorrecting after one bad shot causes wild technique swings; a smart coach adjusts correction size based on recent misses, like Adam does per parameter.

Batch, Stochastic, and Mini-Batch Variants

Batch gradient descent computes the gradient using the entire training set before each update, giving a smooth, accurate descent direction but requiring a full pass over the data per step, which is expensive for large datasets. Stochastic gradient descent (SGD) updates parameters using a single randomly chosen example at a time, making each step fast but noisy, since the gradient estimate is based on just one data point. Mini-batch gradient descent strikes a practical middle ground, computing the gradient over a small batch (commonly 32-256 examples), balancing computational efficiency with a reasonably stable descent direction; it is the de facto standard for training neural networks.

🏏

Cricket analogy: Analyzing an entire team's season data before adjusting one tactic gives a smooth, accurate read but takes forever; reacting after a single ball is fast but noisy; reviewing after each over, like mini-batch, balances speed and stability — the standard approach most coaches use.

python

import numpy as np

def gradient_descent(X, y, lr=0.01, n_iters=1000):
    n_samples, n_features = X.shape
    weights = np.zeros(n_features)
    bias = 0.0

    for _ in range(n_iters):
        y_pred = X @ weights + bias
        error = y_pred - y

        # Gradients of mean squared error w.r.t. weights and bias
        dw = (2 / n_samples) * (X.T @ error)
        db = (2 / n_samples) * np.sum(error)

        weights -= lr * dw
        bias -= lr * db

    return weights, bias

# Usage: weights, bias = gradient_descent(X_train, y_train, lr=0.01, n_iters=1000)

Think of gradient descent as walking downhill in dense fog: you can't see the whole landscape, but you can feel which direction slopes downward under your feet (the gradient) and take a step that size (the learning rate) in that direction, repeating until you reach a valley.

Gradient descent on a non-convex loss surface (typical of neural networks) can get stuck in local minima or saddle points. In practice, techniques like momentum, adaptive learning rates (Adam), and random weight initialization help escape these regions, and empirically most local minima in large networks are nearly as good as the global minimum.

Feature Scaling and Convergence Speed

When input features have very different scales, the loss surface becomes elongated and elliptical rather than roughly circular, causing gradient descent to zigzag inefficiently toward the minimum. Scaling features (e.g. with standardization) reshapes the loss surface to be more symmetric, which typically allows a larger learning rate and much faster, more stable convergence — this is one of the strongest practical reasons feature scaling matters for gradient-based models.

🏏

Cricket analogy: Comparing a batsman's runs (in hundreds) directly against their strike rate (in tens) skews analysis toward the bigger-scaled number; standardizing both stats to comparable scales, like team analysts do, prevents the model from zigzagging toward the wrong conclusion.

Gradient descent iteratively updates parameters in the direction of steepest loss decrease: theta = theta - alpha * gradient.
The learning rate alpha controls step size; too small is slow, too large can diverge or oscillate.
Batch GD uses the full dataset per step, SGD uses one example, and mini-batch GD (the common default) uses small batches.
Adaptive optimizers like Adam adjust per-parameter learning rates based on gradient history.
Non-convex loss surfaces can trap gradient descent in local minima or saddle points; momentum and good initialization help.
Feature scaling reshapes the loss surface and substantially speeds up convergence.

Practice what you learned

1. What does the gradient descent update rule theta = theta - alpha * gradient represent?

2. What is the main tradeoff between batch gradient descent and stochastic gradient descent (SGD)?

3. What typically happens if the learning rate is set far too high?

4. Why does feature scaling generally speed up gradient descent convergence?

5. What advantage do adaptive optimizers like Adam offer over plain SGD?

Was this page helpful?

Hyperparameter Tuning

Topics covered

#Python#MachineLearningBasicsStudyNotes#MachineLearning#GradientDescentExplained#Gradient#Descent#Explained#Update#StudyNotes#SkillVeris

Gradient Descent Explained