Transformers Architecture Cheat Sheet
A cheat sheet for the Transformer architecture covering self-attention, multi-head attention, positional encoding, and Hugging Face model usage.
3 PagesAdvancedMar 2, 2026
Scaled Dot-Product Attention
The core attention computation from first principles.
python
import torchimport torch.nn.functional as Fdef scaled_dot_product_attention(Q, K, V, mask=None): d_k = Q.size(-1) scores = Q @ K.transpose(-2, -1) / (d_k ** 0.5) # scale by sqrt(d_k) if mask is not None: scores = scores.masked_fill(mask == 0, float('-inf')) weights = F.softmax(scores, dim=-1) return weights @ V, weights
Multi-Head Attention
Using PyTorch's built-in attention module.
python
import torch.nn as nnmha = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)attn_output, attn_weights = mha(query, key, value) # self-attention: query = key = value
Hugging Face Usage
Load a pretrained transformer and run inference.
python
from transformers import AutoTokenizer, AutoModeltokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')model = AutoModel.from_pretrained('bert-base-uncased')inputs = tokenizer('Hello world', return_tensors='pt')outputs = model(**inputs)last_hidden_state = outputs.last_hidden_state # shape: (batch, seq_len, hidden_size)
Key Concepts
Core theory behind the Transformer architecture.
- Self-attention- Each token computes a weighted sum over all other tokens, letting the model relate any two positions directly
- Query, Key, Value- Learned linear projections of the input used to compute attention scores and weighted outputs
- Multi-head attention- Runs several attention operations in parallel on different learned subspaces, then concatenates the results
- Positional encoding- Injects order information into token embeddings since attention itself is permutation-invariant
- Feed-forward network- Position-wise MLP applied after attention within each encoder/decoder block
- Residual connections & layer norm- Stabilize training and let gradients flow cleanly through very deep stacks
Pro Tip
Attention has quadratic time and memory complexity in sequence length because every token attends to every other token — for long contexts, look at memory-efficient exact implementations like FlashAttention or sparse/linear attention approximations.
Was this cheat sheet helpful?
Explore Topics
#TransformersArchitecture#TransformersArchitectureCheatSheet#DataScience#Advanced#Scaled#Dot#Product#Attention#MachineLearning#CheatSheet#SkillVeris
