LLM
A Large Language Model (LLM) is a neural network, typically based on the transformer architecture, trained on massive amounts of text to predict and generate human-like language. With billions to hundreds of billions of parameters, LLMs…
Definition
A Large Language Model (LLM) is a neural network, typically based on the transformer architecture, trained on massive amounts of text to predict and generate human-like language. With billions to hundreds of billions of parameters, LLMs learn statistical patterns of grammar, facts, and reasoning from their training data, enabling them to perform tasks like answering questions, writing, summarizing, translating, and coding.
Overview
LLMs are trained in stages. Pretraining exposes the model to enormous volumes of text scraped from books, websites, and other sources, teaching it to predict the next token (a word or word-fragment) given preceding context. This self-supervised process, requiring no manual labeling, is what lets LLMs absorb broad knowledge of language, facts, and reasoning patterns purely from statistical regularities in text. After pretraining, most LLMs undergo further fine-tuning stages: instruction tuning teaches the model to follow user instructions rather than just continue text, and reinforcement learning from human feedback (RLHF) further aligns the model's responses with human preferences for helpfulness, harmlessness, and honesty. These post-training steps are what turn a raw text predictor into a useful assistant. Architecturally, most modern LLMs use the transformer, which relies on a self-attention mechanism to weigh the relevance of every other token in the input when processing each token, allowing the model to capture long-range dependencies in text far better than earlier recurrent architectures. Generation happens autoregressively: the model predicts one token at a time, appending each prediction to the context before predicting the next. LLMs are characterized by a context window (the amount of text they can consider at once), a parameter count (roughly correlating with capacity, though not the only factor in quality), and emergent capabilities that appear at scale, such as few-shot learning and multi-step reasoning. Despite their fluency, LLMs do not 'know' facts in a database sense — they generate statistically plausible continuations, which is why they can hallucinate confidently incorrect information. Modern LLM applications often pair the model with external tools, retrieval systems (RAG), and structured prompting to mitigate these limitations.
Key Concepts
- Built on the transformer architecture using self-attention over input tokens
- Trained via next-token prediction on massive, broad text corpora (pretraining)
- Refined through instruction tuning and RLHF to follow instructions and align with human preferences
- Generates text autoregressively, one token at a time
- Characterized by parameter count, context window size, and training data cutoff
- Exhibits emergent capabilities like few-shot learning and multi-step reasoning at scale
- Prone to hallucination since it generates plausible text rather than retrieving verified facts
- Extensible via prompting, fine-tuning, retrieval augmentation, and tool/function calling