RLHF (reinforcement learning from human feedback)

RLHF (reinforcement learning from human feedback) is a training step that turns a raw language model into an assistant that follows instructions. People compare its answers and pick the best one. A second model learns from those choices what people prefer, and the language model is then trained toward answers that this second model rates highly. OpenAI trained ChatGPT this way. It also helps explain why chat assistants sometimes flatter.

Why is it needed?

A base large language model is trained to predict the next token of text from the internet. Ouyang et al. (OpenAI, 2022) note that this goal differs from "follow the user's instructions helpfully and safely", and that making models bigger does not by itself make them better at following what users want. Their InstructGPT models were trained with human feedback. Labelers preferred the answers of the 1.3-billion-parameter InstructGPT to those of the 175-billion-parameter GPT-3, and on closed-domain tasks such as summaries it made up information in 21% of cases against 41%. In November 2022 OpenAI trained ChatGPT with "the same methods as InstructGPT".

The idea is older. Christiano et al. (2017) taught agents Atari games and simulated robot movements from people's preferences between pairs of short clips of their behaviour, with feedback on less than 1% of the agent's interactions.

How does it work?

InstructGPT used three steps:

1. Supervised fine-tuning. Labelers write good answers for about 13,000 prompts, and the model is trained on those examples.
2. Reward model. The model writes four to nine answers per prompt and people rank them; a separate model learns to predict those rankings. OpenAI collected rankings for 33,000 prompts from a team of about 40 contractors.
3. Reinforcement learning. The model answers new prompts, the reward model scores the answers, and the PPO algorithm (proximal policy optimization) shifts the model toward higher scores. A penalty for drifting too far from the model of step 1 (the KL penalty) keeps it in check; without it, Hugging Face explains, the model can start writing gibberish that fools the reward model.

Newer methods simplify this. DPO (direct preference optimization, Rafailov et al., 2023) skips the reward model and the reinforcement learning and trains directly on pairs of a preferred and a rejected answer; in the authors' tests it matched or beat PPO-based RLHF. Constitutional AI from Anthropic lets an AI model judge harmlessness against written principles (RLAIF).

Where does it go wrong?

• The reward model is a stand-in for people. Optimising it too hard lowers real quality, in line with Goodhart's law (Gao et al., 2022), a form of reward hacking.
• Longer looks better. At ChatGPT's launch OpenAI wrote that the model was often excessively verbose, partly because trainers prefer longer answers that look more comprehensive. Singhal et al. (2023) found that a reward for length alone reproduced most of the gains of RLHF in their tests.
• Agreeing looks better. Sharma et al. (2023) found sycophancy in five AI assistants, and both humans and preference models sometimes preferred convincingly written sycophantic answers to correct ones. In April 2025 OpenAI rolled back a GPT-4o update after an extra reward signal from thumbs-up and thumbs-down feedback had weakened the signal that kept sycophancy in check (OpenAI).
• Other skills can suffer. InstructGPT scored lower than GPT-3 on some public benchmarks, an "alignment tax" that mixing in pre-training updates greatly reduced. Bai et al. (Anthropic, 2022) saw that tax in models below about 10 billion parameters, while their 13B and 52B models did better on zero-shot tests after RLHF.
• Mistakes remain. InstructGPT still made up facts, accepted false premises and gave long, hedging answers to simple questions.