LLaVA
LLaVA (Large Language and Vision Assistant) is an open-source multimodal model that connects a vision encoder to a large language model, enabling visual instruction-following and chat about image content.
Definition
LLaVA (Large Language and Vision Assistant) is an open-source multimodal model that connects a vision encoder to a large language model, enabling visual instruction-following and chat about image content.
Overview
LLaVA was introduced by researchers at the University of Wisconsin-Madison, Microsoft Research, and Columbia University in 2023, with the goal of extending instruction-tuning — the technique that made models like ChatGPT good at following natural-language instructions — into the multimodal domain. It connects a pretrained CLIP vision encoder to a large language model (originally Vicuna, a LLaMA-based chat model) via a simple linear projection layer that maps image features into the language model's embedding space, allowing the LLM to reason over both visual and textual tokens jointly. A key contribution of the original LLaVA paper was a method for generating multimodal instruction-following training data using GPT-4 (used in a text-only capacity, fed detailed image descriptions and bounding box data), producing conversational, detailed-description, and complex-reasoning examples about images without requiring costly human annotation at scale. This data was used to instruction-tune the connected vision-language model, resulting in a system capable of chatting about image content, answering visual questions, and following open-ended visual instructions with a quality that was competitive with proprietary multimodal systems at the time on several benchmarks. LLaVA has since gone through multiple iterations — including LLaVA-1.5, which improved the vision-language connector and training recipe, and LLaVA-NeXT (also called LLaVA-1.6), which improved image resolution handling and reasoning — and has become one of the most widely used and cited open-source baselines for building and researching vision-language chat models. Its relatively simple architecture (a frozen or lightly-tuned vision encoder plus a projection layer feeding into an LLM) has been widely reused and adapted in follow-up multimodal model research across academia and industry.
Key Concepts
- Connects a CLIP vision encoder to an LLM via a lightweight projection layer
- Instruction-tuned on GPT-4-generated multimodal conversation data
- Supports visual chat, question answering, and complex image reasoning
- Open-source weights and training code
- Iterated through LLaVA-1.5 and LLaVA-NeXT (1.6) with improved capabilities
- Widely used as a research baseline for vision-language models