Cross-Entropy Loss
Cross-entropy loss is a loss function that measures the difference between a predicted probability distribution and the true (target) distribution, commonly used to train classification models.
Definition
Cross-entropy loss is a loss function that measures the difference between a predicted probability distribution and the true (target) distribution, commonly used to train classification models.
Overview
Cross-entropy loss is derived from information theory, where cross-entropy measures the average number of bits needed to represent events drawn from a true distribution when using a code optimized for a different, predicted distribution. In machine learning, this translates into a loss function that penalizes a model heavily when it assigns low probability to the correct class, and lightly when it assigns high probability to the correct class — making it a natural fit for training probabilistic classifiers. For binary classification, the loss is typically called binary cross-entropy (or log loss), applied to the output of a sigmoid activation. For multi-class classification, categorical cross-entropy is applied to the output of a softmax function, which converts raw model outputs (logits) into a probability distribution over classes; in this setting the loss simplifies to the negative log of the predicted probability assigned to the correct class. Because of this logarithmic penalty, cross-entropy loss produces especially large gradients when the model is confidently wrong, which drives fast correction during training — a key reason it outperforms alternatives like squared error for classification tasks with probabilistic outputs. Cross-entropy loss (often combined into a single "softmax cross-entropy" or "log-softmax + negative log-likelihood" operation for numerical stability) is the standard training objective for image classifiers, and is the direct ancestor of the "next-token prediction" loss used to pretrain large language models, where the model is trained to assign high probability to the actual next token in a sequence across a huge vocabulary. Label smoothing, a common regularization technique, modifies the target distribution slightly (assigning a small probability mass to incorrect classes) to prevent the model from becoming overconfident.
Key Concepts
- Measures the divergence between predicted and true probability distributions
- Binary cross-entropy applies to sigmoid outputs for two-class problems
- Categorical cross-entropy applies to softmax outputs for multi-class problems
- Penalizes confident wrong predictions much more heavily than uncertain ones
- Produces large gradients when predictions are confidently incorrect, speeding correction
- Standard loss function for image classification and language model pretraining
- Often numerically combined with log-softmax for stable computation
- Compatible with label smoothing to prevent overconfident predictions