CLIP (model)
CLIP (Contrastive Language-Image Pretraining) is a neural network from OpenAI that learns joint representations of images and text by training on paired image-caption data, enabling zero-shot image classification and cross-modal retrieval…
Definition
CLIP (Contrastive Language-Image Pretraining) is a neural network from OpenAI that learns joint representations of images and text by training on paired image-caption data, enabling zero-shot image classification and cross-modal retrieval via natural-language prompts.
Overview
CLIP was released by OpenAI in 2021 and trained on roughly 400 million image-text pairs collected from the internet, using a contrastive objective: an image encoder and a text encoder are trained together so that the embedding of an image and the embedding of its correct caption end up close together in a shared vector space, while embeddings of mismatched image-text pairs are pushed apart. This is a simpler and more scalable supervision signal than manually labeled classification datasets, since captioned images are abundant on the web. The defining capability that resulted from this training is zero-shot classification: instead of training a classifier head on a fixed label set, you can classify an image by comparing it against the text embeddings of candidate class descriptions ("a photo of a dog," "a photo of a cat") and picking the closest match, with no task-specific training required. This made CLIP one of the first vision models to demonstrate GPT-3-style zero-shot generalization in the image domain, and it performed competitively with fully supervised ImageNet classifiers despite never being trained on ImageNet labels. CLIP's dual image/text encoder architecture — typically a Vision Transformer or ResNet for images paired with a Transformer for text — made it immediately useful as a building block rather than an end product. It became the backbone for text-to-image retrieval systems, the guidance signal in early text-to-image generation pipelines (including early DALL-E-adjacent and diffusion-guided systems before dedicated text encoders like those in Stable Diffusion took over), content moderation classifiers, and image search engines. CLIP's open release (weights and training code) made it one of the most widely reused multimodal models in the open-source ecosystem; variants and re-trainings such as OpenCLIP (LAION) extended CLIP to larger datasets and model sizes, and CLIP-style contrastive pretraining influenced later multimodal models including ImageBind, which extends the same contrastive idea across additional modalities like audio and depth.
Key Concepts
- Joint image-text embedding space trained via contrastive learning
- Zero-shot image classification using natural-language class descriptions
- Trained on roughly 400 million internet image-caption pairs
- Dual-encoder architecture (image encoder + text encoder)
- No task-specific fine-tuning required for many classification and retrieval tasks
- Open-sourced weights and training code, spawning community variants like OpenCLIP
- Used as a guidance/scoring signal in early text-to-image generation systems
- Enables cross-modal search: text-to-image and image-to-text retrieval