Learning Rate Scheduling
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up convergence and improve final model quality.
5 resources across 1 library
Glossary Terms(5)
Batch Normalization
Batch normalization is a neural network training technique that normalizes the inputs to each layer using the mean and variance computed over a mini-batch, the…
Layer Normalization
Layer normalization is a neural network normalization technique that normalizes activations across the features of a single training example, rather than acros…
Learning Rate Scheduling
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up converg…
Adam Optimizer
Adam (Adaptive Moment Estimation) is a gradient-based optimization algorithm that maintains per-parameter adaptive learning rates using running estimates of th…
Stochastic Gradient Descent
Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of the loss computed on a small, rand…