Florence-2
Florence-2 is a lightweight vision foundation model from Microsoft that uses a unified, prompt-based sequence-to-sequence architecture to perform a wide range of vision and vision-language tasks — captioning, object detection,…
Definition
Florence-2 is a lightweight vision foundation model from Microsoft that uses a unified, prompt-based sequence-to-sequence architecture to perform a wide range of vision and vision-language tasks — captioning, object detection, segmentation, and OCR — from a single set of weights.
Overview
Florence-2 departs from the trend of ever-larger vision models by showing that a compact encoder-decoder architecture, trained on a very large and densely annotated dataset (FLD-5B, roughly 5.4 billion annotations over 126 million images), can match or beat much bigger specialist models across many vision tasks. Instead of separate heads for detection, segmentation, and captioning, Florence-2 casts every task as a text-generation problem: a task-specific prompt (like '<OD>' for object detection or '<CAPTION>' for captioning) is fed alongside the image, and the model generates a structured text sequence representing the answer, including bounding-box coordinates as text tokens. The architecture pairs a DaViT vision encoder with a standard transformer encoder-decoder, keeping total parameter counts small (base and large variants at 0.23B and 0.77B parameters) so the model runs efficiently on modest hardware and fine-tunes quickly for downstream tasks. This unification is what makes Florence-2 attractive for production pipelines: a single checkpoint can replace several separate models for captioning, grounding, region proposal, and dense region captioning. Microsoft released Florence-2 openly on Hugging Face under a permissive license, and it has become a common building block in multimodal pipelines — used for automatic data labeling, generating grounded captions to train other vision-language models, and as a fast pre-processing step in document and image understanding stacks. Its small footprint and multi-task versatility distinguish it from larger general-purpose vision-language models like LLaVA or GPT-4V, which trade efficiency for broader open-ended reasoning ability.
Key Concepts
- Single unified model handles captioning, detection, segmentation, grounding, and OCR
- Prompt-based task specification using special task tokens
- Trained on FLD-5B, a 5.4B-annotation, 126M-image dataset
- Compact sizes (0.23B base, 0.77B large) for fast inference and fine-tuning
- DaViT vision encoder paired with a transformer encoder-decoder
- Outputs structured text sequences, including bounding boxes as tokens
- Open weights released on Hugging Face under MIT license
- Strong zero-shot performance relative to model size