Reinforcement learning (RL)

Reinforcement learning (RL) trains AI through trial and error. The system tries an action, sees what happens and gets a score, a reward or a penalty, and over many attempts it learns which choices pay off in the long run. It taught a computer to beat a Go world champion, and it is part of how chatbots such as ChatGPT learned to give answers that people prefer.

How does it work?

In the textbook by Sutton and Barto, reinforcement learning means learning what to do in each situation so as to maximise a numerical reward. Nobody tells the learner which action is right: it has to find out by trying. OpenAI's introduction Spinning Up describes the loop. An agent observes the state of its environment and chooses an action; the environment changes and returns a reward, a number for how good or bad the new state is. The agent aims for the highest total reward over time, and the rule it uses to choose actions is called its policy.

Sutton and Barto name two features that set RL apart: trial and error, and delayed reward, because an action can affect every reward that follows. Unlike supervised learning, there are no labelled examples of the right action. The agent also faces a dilemma: it has to exploit what already works to earn reward, and explore untried actions to find something better.

Which methods are there?

Spinning Up sorts the algorithms by what they learn. Q-learning methods, such as DQN, estimate how much future reward each action is worth in a given state. Policy optimisation methods adjust the policy itself; PPO (Schulman et al., 2017) is one of them. According to Spinning Up, Q-learning makes better use of its data when it works, and policy optimisation tends to be more stable.

How do language models use it?

In OpenAI's InstructGPT work (Ouyang et al., 2022), people ranked a model's answers, a reward model learned to predict their preference, and PPO tuned the language model to score well on that reward model. This is RLHF (reinforcement learning from human feedback), and OpenAI trained ChatGPT (November 2022) with the same method. The InstructGPT authors note that it aligns a model with the preferences of a specific group of people, mostly their labelers and researchers.

Reasoning models also get rewards that a program can check. DeepSeek trained DeepSeek-R1-Zero with RL alone, using the GRPO algorithm: a maths answer earned a reward when it matched the known answer, and code when it passed test cases (DeepSeek-AI, 2025).

Where does it go wrong?

• The agent optimises the reward as written. DeepMind calls this specification gaming and had collected around 60 examples by April 2020. In one, an agent was rewarded for the height of the bottom of a red Lego block that it should stack on a blue one; it flipped the red block over instead.
• A learned reward model can be exploited too, which is known as reward hacking. For that reason DeepSeek kept such models out of its training for reasoning tasks.

Example

DeepMind's AlphaGo first studied thousands of expert games and then played against versions of itself. In March 2016 it beat Lee Sedol 4-1 in Seoul. Its successor AlphaGo Zero skipped the human games. It started from random play and received its reward at the end of each game: +1 for a win, -1 for a loss. In about three days of training it played 4.9 million games against itself, and it then beat the version that had defeated Lee Sedol by 100 games to 0 (Silver et al., 2017).