Attention Mechanism

Whisper

Whisper is an open-source automatic speech recognition (ASR) model from OpenAI that transcribes and translates spoken audio into text across dozens of language…

PaLM

PaLM (Pathways Language Model) is a 540-billion-parameter dense large language model developed by Google Research and announced in April 2022, trained using Go…

BERT

BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only Transformer language model released by Google AI in 2018 that learns deep bid…

RoBERTa

RoBERTa (Robustly Optimized BERT Pretraining Approach) is an encoder-only language model released by Facebook AI Research in 2019 that improves on BERT by refi…

T5

T5 (Text-to-Text Transfer Transformer) is an encoder-decoder Transformer model introduced by Google Research in 2019 that reframes every NLP task — translation…

GPT-2

GPT-2 is a decoder-only Transformer-based language model released by OpenAI in 2019, notable for its 1.5-billion-parameter scale and for its unusually staged p…

AlphaFold

AlphaFold is a deep learning system developed by Google DeepMind that predicts a protein's three-dimensional structure from its amino acid sequence with accura…

Parti

Parti (Pathways Autoregressive Text-to-Image model) is a text-to-image generation model from Google Research, introduced in 2022, that treats image generation…

CLIP

CLIP (Contrastive Language-Image Pretraining) is a neural network released by OpenAI in 2021 that learns to connect images and natural-language text by trainin…

Wav2Vec

Wav2Vec is a family of self-supervised speech representation learning models developed by Facebook AI Research (Meta AI), which learn useful audio representati…

Vision Transformer (ViT)

Vision Transformer (ViT) is an image classification architecture introduced by Google Research in 2020 that applies the Transformer architecture — originally d…

Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural network architecture design in which a model is composed of many specialized sub-networks ("experts"), with a learned rout…

Multimodal Model

A multimodal model is an AI system that can process and often generate more than one type of data — such as text, images, audio, or video — within a single uni…

Embedding Model

An embedding model converts text, images, or other data into dense numerical vectors that capture semantic meaning, so that items with similar meaning end up c…

Speech-to-Text Model

A speech-to-text (STT) model, also called automatic speech recognition (ASR), converts spoken audio into written text, transcribing what was said so it can be…

Diffusion Transformer (DiT)

A Diffusion Transformer (DiT) is a generative architecture that replaces the convolutional U-Net backbone traditionally used in diffusion models with a Transfo…

State Space Model (Mamba)

A State Space Model (SSM), in the context of modern sequence modeling, is a neural architecture — with Mamba as a prominent example — that processes sequences…

Sparse Mixture Model

A sparse mixture model is a neural network design in which only a small subset of the model's parameters are activated for any given input, rather than the ent…

Vision-Language Model (VLM)

A Vision-Language Model (VLM) is an AI model trained to jointly understand images and text, enabling tasks such as describing an image in words, answering ques…

Chain of Thought

Chain of thought is a prompting technique that encourages a large language model to generate intermediate reasoning steps before producing a final answer, impr…

Backpropagation

Backpropagation is the algorithm used to train neural networks by efficiently computing how much each parameter contributed to the model's error, so those para…

Attention Mechanism

The attention mechanism is a neural network technique that allows a model to dynamically weigh the importance of different parts of its input when producing ea…

Tokenization

Tokenization is the process of breaking text into smaller units, called tokens, that a language model can process — typically words, subwords, or characters.

Natural Language Processing (NLP)

Natural Language Processing (NLP) is the field of artificial intelligence focused on enabling computers to understand, interpret, generate, and interact using…