Stochastic Gradient Descent
Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of the loss computed on a small, randomly sampled batch of training data rather than the entire dataset at once.
Definition
Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of the loss computed on a small, randomly sampled batch of training data rather than the entire dataset at once.
Overview
Gradient descent is the foundational algorithm for training most machine learning models: it repeatedly adjusts parameters in the direction that most reduces a loss function, based on the loss function's gradient. Computing that gradient over an entire training dataset ("batch" gradient descent) is accurate but computationally expensive and memory-intensive for large datasets. Stochastic gradient descent instead estimates the gradient using a single randomly chosen example, or more commonly in practice, a small "mini-batch" of examples, updating parameters far more frequently at the cost of noisier gradient estimates. This noise turns out to be beneficial as well as necessary for scalability: the randomness helps the optimizer escape shallow local minima and saddle points that a smoother, fully accurate gradient might settle into, and mini-batch training makes it possible to train on datasets and models far too large to fit gradient computations over in one pass. A single full pass through the training data is called an epoch, and training typically runs for many epochs, reshuffling the data each time. Plain SGD is often enhanced with momentum, which accumulates an exponentially weighted average of past gradients to smooth out updates and accelerate progress along consistent directions, damping oscillation in narrow valleys of the loss surface. Nesterov momentum is a further refinement that looks ahead along the current trajectory before computing the gradient. While adaptive optimizers like Adam have become the default in many domains, particularly for training transformers, SGD with momentum remains widely used and sometimes preferred in computer vision, where it can produce models that generalize slightly better, and it underlies the theoretical foundations most other optimizers build on.
Key Concepts
- Updates parameters using gradients estimated from mini-batches rather than the full dataset
- Far more computationally and memory efficient than full-batch gradient descent
- Gradient noise from sampling helps escape shallow local minima and saddle points
- Commonly enhanced with momentum to smooth and accelerate updates
- Nesterov momentum looks ahead along the trajectory before computing gradients
- Requires a learning rate hyperparameter, often paired with a decay schedule
- Training proceeds over multiple epochs, each a full pass through shuffled data
- Forms the theoretical and practical basis for adaptive optimizers like Adam