CLIP
By OpenAI
CLIP (Contrastive Language-Image Pretraining) is a neural network released by OpenAI in 2021 that learns to connect images and natural-language text by training on hundreds of millions of image-caption pairs, enabling it to judge how well…
Definition
CLIP (Contrastive Language-Image Pretraining) is a neural network released by OpenAI in 2021 that learns to connect images and natural-language text by training on hundreds of millions of image-caption pairs, enabling it to judge how well an arbitrary image matches an arbitrary text description without task-specific fine-tuning.
Overview
CLIP consists of two encoders trained jointly: an image encoder (a Vision Transformer or ResNet variant) that converts an image into a vector embedding, and a text encoder (a Transformer) that converts a caption into a vector embedding in the same shared space. During training, CLIP is shown batches of image-text pairs scraped from the web and learns, via a contrastive objective, to pull the embeddings of matching image-caption pairs close together while pushing non-matching pairs apart — without ever being told explicit class labels like "cat" or "dog." This training approach gives CLIP strong zero-shot image classification ability: to classify an image into one of several candidate categories, CLIP simply compares the image embedding against text embeddings of prompts like "a photo of a cat" for each candidate label and picks the closest match, often rivaling models trained specifically for that classification task. Because CLIP learns a general notion of semantic similarity between images and text rather than a fixed label set, it generalizes to categories and concepts it was never explicitly trained to recognize. CLIP's biggest downstream impact came through its use as a component in other systems rather than as a standalone classifier: its text-image similarity scoring became a key building block in guiding early text-to-image diffusion models toward prompt-faithful outputs, and CLIP-style text encoders are used inside models such as Stable Diffusion and Stable Diffusion XL to translate prompts into a form the image generator can condition on. CLIP also underlies many image search, content moderation, and multimodal retrieval systems built since its release.
Key Concepts
- Joint image and text encoders trained into a shared embedding space
- Trained via a contrastive objective on hundreds of millions of image-caption pairs from the web
- Enables zero-shot image classification without task-specific fine-tuning
- No fixed label set — generalizes to novel categories described in natural language
- Widely used as the text-conditioning component inside text-to-image diffusion models
- Powers multimodal semantic search and content moderation systems