Gradient Accumulation
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-batches before performing a single optimizer update.
2 resources across 1 library
Glossary Terms(2)
Gradient Clipping
Gradient clipping is a training technique that caps the magnitude of gradients during backpropagation, preventing the exploding-gradient problem that can desta…
Gradient Accumulation
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-b…