CogVLM
CogVLM is an open-source visual language model developed by researchers at Zhipu AI and Tsinghua University that combines a frozen pretrained language model with a trainable 'visual expert' module inserted into each transformer layer,…
Definition
CogVLM is an open-source visual language model developed by researchers at Zhipu AI and Tsinghua University that combines a frozen pretrained language model with a trainable 'visual expert' module inserted into each transformer layer, enabling deep fusion of vision and language features without degrading the language model's original text capabilities.
Overview
CogVLM, introduced in a 2023 paper, addresses a common weakness in many vision-language models of that era: shallow, adapter-based fusion of image and text features that often caused a measurable drop in the underlying language model's text-only performance. Instead of only aligning image embeddings to a frozen LLM's input space via a lightweight projection layer, CogVLM adds a trainable 'visual expert module' — a parallel set of QKV and MLP weights — inside each layer of the language model, specifically for processing visual tokens, while keeping the original language model weights frozen. This gives the model a deep, layer-by-layer pathway for visual reasoning while preserving the base LLM's generative and language capabilities. CogVLM uses a vision transformer (EVA2-CLIP) as its image encoder, and pairs the resulting visual features with a Vicuna-based (LLaMA-derived) language model backbone. At the time of release, CogVLM achieved state-of-the-art or near-state-of-the-art results on multiple cross-modal benchmarks including visual question answering, image captioning, and visual grounding, outperforming several larger closed models on some tasks. The team also released CogAgent, a variant specialized for GUI understanding and agent tasks such as reading and interacting with screenshots of web pages and applications, and CogVLM2, an updated version with improved resolution support and instruction-following. As an open-weight model, CogVLM is widely used in academic research and by developers building custom multimodal applications who want to fine-tune or self-host a capable vision-language model rather than depend on a closed API, and it has served as a reference architecture for later work on deep vision-language fusion.
Key Concepts
- Trainable 'visual expert module' added to every transformer layer for deep fusion
- Keeps the base language model weights frozen, preserving text-only performance
- Uses EVA2-CLIP vision encoder paired with a Vicuna/LLaMA-based LLM backbone
- Strong results on VQA, captioning, and visual grounding benchmarks at release
- Open-weight, self-hostable, and fine-tunable
- CogAgent variant specialized for GUI/screenshot understanding and agentic tasks
- CogVLM2 successor adds higher resolution and improved instruction following