What is the Transformer architecture?

The Transformer is a neural network architecture based entirely on self-attention, allowing it to model relationships between all tokens in a sequence in parallel rather than one at a time, as recurrent models do.

Who introduced the Transformer?

The Transformer was introduced by Vaswani et al. at Google in the 2017 paper 'Attention Is All You Need.'

What is self-attention?

Self-attention is a mechanism that lets every token in a sequence compute a weighted representation based on its relevance to every other token, regardless of their distance apart in the sequence.

Why did Transformers replace RNNs and LSTMs?

Transformers process all tokens in parallel rather than sequentially, making them far more efficient to train on modern hardware, and self-attention captures long-range dependencies more effectively than recurrent architectures.

What is the difference between encoder-only, decoder-only, and encoder-decoder Transformers?

Encoder-only models like BERT are optimized for understanding input, decoder-only models like GPT are optimized for generating text, and encoder-decoder models handle tasks like translation that map one sequence to another.

What is a limitation of the Transformer architecture?

Standard self-attention computation scales quadratically with sequence length, making very long context windows computationally expensive, which has motivated research into more efficient attention variants.

Are Transformers used outside of NLP?

Yes, Transformers are widely used in computer vision through Vision Transformers, as well as in speech processing, multimodal models, and even scientific applications like protein structure prediction.

AI Fundamentals

Transformer Architecture

AdvancedConcept9.1K learners

The Transformer is a neural network architecture built entirely around the self-attention mechanism, allowing it to model relationships between all elements of an input sequence in parallel rather than processing them one at a time.

#TransformerArchitecture #AIFundamentals#Concept#Advanced#EncoderDecoderModel #SequenceToSequenceModel #ContextWindow #AttentionMechanism #ArtificialIntelligence#Glossary#SkillVeris

Definition

Overview

Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., the Transformer replaced the recurrent and convolutional architectures that previously dominated sequence modeling, such as LSTMs and RNNs, which processed tokens sequentially and struggled to capture long-range dependencies efficiently. Instead, the Transformer relies on self-attention, a mechanism that lets every token in a sequence directly attend to every other token, computing a weighted representation based on relevance regardless of distance in the sequence. This parallelizability made Transformers dramatically more efficient to train on modern hardware like GPUs and TPUs, which was a key enabler of the scaling that later produced large language models. The original architecture consists of an encoder stack and a decoder stack, each built from repeated layers combining multi-head self-attention (which lets the model attend to different types of relationships simultaneously through parallel attention "heads") with position-wise feedforward networks, residual connections, and layer normalization. Because self-attention has no inherent sense of token order, positional encodings are added to the input embeddings to inject sequence position information. Many modern LLMs use only the decoder portion (decoder-only architectures like GPT), or only the encoder portion (encoder-only architectures like BERT), depending on whether the task is generative or primarily about understanding input. The Transformer's efficiency and expressiveness made it the dominant architecture across not just NLP but also computer vision (via Vision Transformers), speech, and multimodal models, effectively replacing task-specific architectures across much of deep learning. Its main computational limitation is that standard self-attention scales quadratically with sequence length, since every token must attend to every other token, which has motivated a large body of research into more efficient attention variants — such as sparse attention, linear attention, and sliding window attention — to support longer context windows without prohibitive compute costs.

Key Concepts

Built entirely around the self-attention mechanism, without recurrence
Processes all tokens in a sequence in parallel rather than sequentially
Multi-head attention captures multiple types of relationships simultaneously
Uses positional encodings to inject sequence order information
Composed of encoder and/or decoder stacks with residual connections and layer normalization
Introduced in "Attention Is All You Need" (Vaswani et al., 2017)
Standard self-attention scales quadratically with sequence length
Foundation for GPT, BERT, Vision Transformers, and most modern LLMs

Use Cases

Powering large language models for text generation and understanding

Machine translation and other sequence-to-sequence NLP tasks

Image classification and object detection via Vision Transformers

Speech recognition and audio processing models

Multimodal models combining text, image, and audio understanding

Protein structure prediction, as in AlphaFold's architecture

Frequently Asked Questions

From the Blog

AI & Technology

How Large Language Models Actually Work

LLMs seem magical until you understand what they are: next-token predictors trained on massive text corpora. This guide explains tokenisation, embeddings, the transformer architecture, attention mechanism, and how training works — without requiring a maths degree.