BERT
By Google AI
BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only Transformer language model released by Google AI in 2018 that learns deep bidirectional representations of text by jointly conditioning on both left and…
Definition
BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only Transformer language model released by Google AI in 2018 that learns deep bidirectional representations of text by jointly conditioning on both left and right context, and was a breakthrough for a wide range of natural language understanding tasks.
Overview
Before BERT, most language models processed text in a single direction — left to right or right to left — which limited how well they could capture the full context of a word within a sentence. BERT's key innovation was training a deep Transformer encoder using a masked language modeling objective: random words in a sentence are hidden, and the model must predict them using context from both directions simultaneously, giving it a much richer understanding of word meaning in context. BERT was pretrained on a large corpus (BooksCorpus and English Wikipedia) using two objectives: masked language modeling and next-sentence prediction, which teaches the model to understand relationships between sentence pairs. After pretraining, BERT could be fine-tuned with a small additional output layer on specific downstream tasks — question answering, sentiment classification, named entity recognition, and more — often surpassing prior task-specific architectures with minimal task-specific engineering. BERT's release popularized the now-standard "pretrain then fine-tune" paradigm in NLP and directly inspired a wave of derivative models, including RoBERTa, DistilBERT, ALBERT, and domain-specific variants like BioBERT and SciBERT. Unlike generative models such as GPT-2 or GPT-3, BERT is not designed to generate free-form text — it is an encoder built to produce rich contextual representations for classification, extraction, and understanding tasks. While large generative LLMs have since become the dominant paradigm for many applications, BERT and its descendants remain widely used in production systems for tasks like search ranking, semantic similarity, and text classification, where smaller, specialized encoder models are often more efficient and cost-effective than general-purpose generative models.
Key Concepts
- Encoder-only Transformer architecture, unlike decoder-based generative models
- Bidirectional context modeling via masked language modeling pretraining
- Next-sentence prediction objective for understanding sentence relationships
- Pretrain-then-fine-tune paradigm requiring minimal task-specific architecture changes
- Released in Base (110M parameters) and Large (340M parameters) configurations
- Trained on BooksCorpus and English Wikipedia
- Spawned a family of derivative models including RoBERTa, DistilBERT, and ALBERT