Contrastive Learning
Representation learning technique
Contrastive learning is a self-supervised representation learning technique that trains a model to produce similar embeddings for semantically related (positive) pairs of examples and dissimilar embeddings for unrelated (negative) pairs.
Definition
Contrastive learning is a self-supervised representation learning technique that trains a model to produce similar embeddings for semantically related (positive) pairs of examples and dissimilar embeddings for unrelated (negative) pairs.
Overview
Contrastive learning learns useful data representations without requiring manually labeled data, by constructing training signal directly from the data's own structure. A common recipe generates positive pairs by applying different random augmentations (such as cropping, color distortion, or rotation for images) to the same underlying example, so the two augmented views should map to similar points in the learned embedding space, while other, unrelated examples in the same batch serve as negative pairs whose embeddings should be pushed apart. The model is trained using a contrastive loss function, most commonly InfoNCE (Noise-Contrastive Estimation), which simultaneously pulls positive pairs together and pushes negative pairs apart in embedding space. Landmark contrastive learning frameworks include SimCLR, which showed that large batch sizes, strong data augmentation, and a learned nonlinear projection head before the contrastive loss substantially improve representation quality; MoCo (Momentum Contrast), which maintains a large, consistent dictionary of negative examples using a slowly updated momentum encoder to decouple the number of negatives from the batch size; and CLIP, which applies contrastive learning across image-text pairs, learning a shared embedding space where matching images and captions are close together, enabling powerful zero-shot image classification and cross-modal search. The representations learned via contrastive pretraining transfer remarkably well to downstream supervised tasks — often matching or exceeding fully supervised pretraining while using no labels during the pretraining phase itself — which has made contrastive learning a cornerstone of modern self-supervised learning pipelines. It is widely used to pretrain vision encoders before fine-tuning on smaller labeled datasets, to build multimodal embedding spaces connecting images, text, audio, or other modalities (as in CLIP and its successors), to power semantic search and recommendation systems via embedding similarity, and as a component within sentence-embedding models used for retrieval-augmented generation and semantic similarity tasks in NLP.
Key Concepts
- Trains embeddings to pull positive pairs together and push negatives apart
- Self-supervised — requires no manually labeled training data
- Commonly uses data augmentation to construct positive pairs
- Trained with contrastive losses like InfoNCE
- SimCLR demonstrated the impact of batch size, augmentation, and projection heads
- MoCo decouples negative sample count from batch size via a momentum encoder
- CLIP extends the technique across image-text pairs for multimodal embeddings
- Learned representations transfer strongly to downstream supervised tasks