GRPO (group relative policy optimization)

GRPO (group relative policy optimization) is a method for training an AI language model further. The model answers the same question several times, each answer gets a score, and the model is nudged towards the answers that scored above the group's average. DeepSeek used it to teach its R1 model to reason. It suits tasks where a program can check the answer, such as sums or code.

How does it work?

DeepSeek introduced GRPO in February 2024 in its DeepSeekMath paper, as a variant of PPO (proximal policy optimization), the reinforcement learning method used in RLHF. Each training step has four parts:

1. For every question, the model writes a group of answers. DeepSeekMath sampled 64 per question.
2. Each answer gets a reward, from a reward model or from a rule that checks the result.
3. The rewards are compared within the group: subtract the group average and divide by the group's standard deviation. The result, the advantage, is positive for answers that beat the average and negative for the rest.
4. The model is updated to make the positive answers more likely and the negative ones less likely. As in PPO, a clipped objective keeps each step small, and a KL term keeps the model close to a reference version of itself.

The saving is in step 3. PPO needs a separate value model to estimate what score to expect for an answer, and DeepSeek describes that model as typically about as large as the model being trained. GRPO uses the group average as that expectation, so it needs no value model; it still needs a reward. In DeepSeekMath, GRPO raised the 7B instruction model from 46.8% to 51.7% on the MATH benchmark and from 82.9% to 88.2% on GSM8K.

Where is it used?

DeepSeek trained DeepSeek-R1-Zero with GRPO directly on its base model, without first training it on example answers. The reward came from rules: is the final answer correct, and is the reasoning between <think> tags? DeepSeek avoided neural reward models because they invite reward hacking at this scale. In the Nature version of the paper (September 2025), R1-Zero's average score on the AIME 2024 maths exam rose from 15.6% to 77.9%.

Training with rewards that a program can check is called RLVR (reinforcement learning with verifiable rewards). Hugging Face's TRL library added a GRPO trainer in January 2025 and made it part of its stable interface in TRL v1.0 (March 2026). The quick start for that version:

1
from datasets import load_dataset
2
from trl import GRPOTrainer
3
from trl.rewards import accuracy_reward
4
 
5
dataset = load_dataset("trl-lib/DeepMath-103K", split="train")
6
 
7
trainer = GRPOTrainer(
8
    model="Qwen/Qwen2-0.5B-Instruct",
9
    reward_funcs=accuracy_reward,
10
    train_dataset=dataset,
11
)
12
trainer.train()

According to the documentation, this run takes about a day on 8 GPUs.

Where does it go wrong?

• No signal from easy or impossible questions. If every answer in a group gets the same reward, every advantage is zero and the question teaches the model nothing (DAPO, March 2025).
• Instability. The DAPO team's first GRPO run on Qwen2.5-32B scored 30 points on AIME, against 47 for DeepSeek's RL recipe on the same base model; they found entropy collapse (the model stops trying different answers), noisy rewards and unstable training, and proposed four fixes.
• Length bias. Liu et al. (March 2025) showed that GRPO's normalisation makes answers longer, especially wrong ones, and proposed a corrected version, Dr. GRPO. TRL v1.0 no longer divides by answer length and leaves out the KL term by default.
• Reward hacking. The model optimises whatever earns reward, flaws included. DeepSeek chose rule-based rewards because neural reward models are easier to exploit at scale.

Example

Suppose a company trains a small open model to turn customer emails into order data. For each email the model writes eight answers, and a script gives 1 point when the JSON contains the right order number. If three answers score 1, those three get a positive advantage and the other five a negative one, and the next version of the model writes more answers like the first three. If all eight score 1, that email adds nothing to the training.