RLHF
Reinforcement Learning from Human Feedback (RLHF) is a training technique used to align language models with human preferences by having people rank or rate model outputs, training a reward model on those preferences, and then using…
Definition
Reinforcement Learning from Human Feedback (RLHF) is a training technique used to align language models with human preferences by having people rank or rate model outputs, training a reward model on those preferences, and then using reinforcement learning to fine-tune the language model to produce outputs that score highly according to that reward model. It is a key step in making raw pretrained models helpful, safe, and aligned with user intent.
Overview
A pretrained language model learns to predict plausible text, but 'plausible' is not the same as 'helpful,' 'safe,' or 'aligned with what a human actually wants.' RLHF was developed to close this gap by directly incorporating human judgment into the training process. The process typically has three stages. First, the pretrained (and usually instruction-tuned) model generates multiple candidate responses to a given prompt. Second, human annotators compare and rank these responses by quality — for helpfulness, accuracy, tone, and safety — and this preference data is used to train a separate reward model that learns to predict which of two responses a human would prefer. Third, the language model is further trained using a reinforcement learning algorithm (commonly a variant of Proximal Policy Optimization, PPO) to generate outputs that the reward model scores highly, while a constraint keeps the updated model from drifting too far from its original behavior to preserve general capability and coherence. RLHF has been central to transforming raw pretrained language models into the conversational, instruction-following assistants widely used today: it teaches models to follow instructions more reliably, avoid generating harmful or offensive content, decline inappropriate requests, express appropriate uncertainty, and generally better match what users actually want rather than just what is statistically likely to follow a prompt. RLHF has known limitations: it depends heavily on the quality, consistency, and diversity of human feedback data, and reward models can be 'gamed' by the policy model finding outputs that score well on the proxy reward without truly reflecting genuine quality — a failure mode called reward hacking. More recent alignment approaches, such as Direct Preference Optimization (DPO) and Constitutional AI, aim to achieve similar alignment goals with different, sometimes simpler or more scalable, training procedures, and RLHF continues to evolve alongside these alternatives.
Key Concepts
- Aligns model behavior with human preferences using human-ranked comparisons of outputs
- Trains a separate reward model to predict human preference between candidate responses
- Uses reinforcement learning (commonly PPO) to optimize the language model against the reward model
- Constrains the fine-tuned model to avoid drifting too far from its original behavior
- Key step in transforming a raw pretrained model into a helpful, safety-conscious assistant
- Vulnerable to reward hacking, where the model optimizes for the proxy reward rather than true quality
- Depends heavily on the quality and diversity of human feedback data collected
- Complemented or alternatively addressed by newer methods like Direct Preference Optimization (DPO)