BLIP
BLIP (Bootstrapping Language-Image Pre-training) is a vision-language pretraining framework developed by Salesforce Research that unifies understanding and generation tasks, and introduces a 'CapFilt' bootstrapping method that uses a…
Definition
BLIP (Bootstrapping Language-Image Pre-training) is a vision-language pretraining framework developed by Salesforce Research that unifies understanding and generation tasks, and introduces a 'CapFilt' bootstrapping method that uses a captioner and filter to clean noisy web-scraped image-text data during training.
Overview
BLIP, published by Salesforce Research in 2022, was designed to address two limitations common in prior vision-language pretraining: model architectures that excelled at either understanding tasks (like image-text retrieval) or generation tasks (like captioning) but not both well, and a heavy reliance on noisy image-alt-text pairs scraped from the web, which limits supervision quality. BLIP introduces a multimodal mixture-of-encoder-decoder architecture that can operate as a unimodal encoder, an image-grounded text encoder, or an image-grounded text decoder, allowing a single model to be trained jointly on three objectives: image-text contrastive learning, image-text matching, and image-conditioned language modeling. The framework's other major contribution is CapFilt (Captioning and Filtering), a dataset bootstrapping procedure: a captioner module generates synthetic captions for web images, and a filter module removes noisy captions (both original web captions and synthetic ones) that don't match the image well, producing a cleaner, higher-quality training corpus without requiring additional human annotation. This self-improving data pipeline allowed BLIP to achieve strong results on captioning, retrieval, and visual question answering benchmarks using web-scale data more effectively than prior methods that used raw scraped pairs directly. BLIP established a widely reused vision-language pretraining recipe and became the architectural predecessor to BLIP-2, which replaced BLIP's end-to-end joint training with a more compute-efficient approach using frozen pretrained image encoders and language models connected by a lightweight Q-Former. BLIP itself remains a commonly used open-source model for image captioning and retrieval tasks, and its CapFilt bootstrapping idea influenced later work on synthetic data generation and filtering for multimodal training.
Key Concepts
- Unified architecture supporting both vision-language understanding and generation
- Trained jointly on contrastive, matching, and language-modeling objectives
- CapFilt bootstrapping pipeline generates and filters synthetic captions
- Improves training-data quality from noisy web-scraped image-text pairs
- Strong performance on captioning, retrieval, and VQA benchmarks
- Open-source, widely used as a lightweight captioning/retrieval model
- Architectural predecessor to BLIP-2