Reinforcement Learning from Human Feedback (RLHF) is a training technique that aligns LLM behavior with human values and preferences by having humans rate model outputs, training a reward model on those ratings, and then fine-tuning the LLM to maximize the reward model's score. RLHF is the key technology that transformed base LLMs — which are essentially next-token predictors with no sense of helpfulness, safety, or instruction-following — into the useful assistants people interact with today. Without RLHF (or its successor techniques like DPO and RLAIF), a raw LLM would respond to "How do I make a cake?" by continuing the text statistically rather than providing a helpful recipe.
Why it matters
RLHF is what makes the difference between a raw language model and a useful AI assistant. It teaches models to be helpful rather than merely plausible, to refuse harmful requests, to acknowledge uncertainty, and to follow instructions precisely. For organizations deploying LLMs, understanding RLHF explains why different models have different "personalities" and safety behaviors — these are direct consequences of the human preference data and reward model used during training. It also explains the phenomenon of reward hacking, where models learn to game the reward signal by producing outputs that score well on the reward model but are not genuinely better for the user, such as being excessively verbose or sycophantic.
How it works
RLHF proceeds in three stages. First, supervised fine-tuning (SFT): the base model is trained on high-quality demonstration data showing ideal assistant behavior. Second, reward model training: human evaluators compare pairs of model outputs (for the same prompt) and indicate which response is better. These preference pairs train a separate reward model that learns to predict human preferences. Third, reinforcement learning: the SFT model generates responses, the reward model scores them, and the language model's weights are updated via Proximal Policy Optimization (PPO) or similar RL algorithms to increase the probability of high-scoring responses. This loop runs for thousands of iterations, gradually specializing the model toward human-preferred behavior while maintaining broad language capabilities.
Example
A model provider wants to improve their LLM's ability to handle math questions honestly — admitting when problems are beyond its reliability instead of guessing confidently. Human evaluators rate pairs of responses to math questions on two criteria: correctness and calibration (does the model express appropriate confidence?). Response A: "The answer is 42" (correct but overconfident). Response B: "I believe the answer is 42, though this involves a multi-step calculation where I could make errors — I'd recommend verifying." Evaluators consistently prefer B. After thousands of such comparisons and RLHF training, the model learns to calibrate its confidence — providing clear answers for simple problems while adding appropriate hedging for complex ones. This reduces user over-reliance on the model for tasks it is likely to get wrong.