Group relative policy optimization (GRPO) is a policy gradient algorithm in reinforcement learning that estimates the advantage of each sampled output by comparing it with other outputs generated for the same prompt, instead of relying on a learned value function. It was introduced in 2024 with the DeepSeekMath work and became widely known as the training method behind the reasoning model DeepSeek-R1.
GRPO is a simplification of proximal policy optimization designed for fine-tuning large language models. By removing the critic network, it cuts memory use and engineering complexity, which makes it a common choice for reinforcement learning with verifiable rewards, such as math answers or code that must pass tests.
How group relative policy optimization computes advantages
GRPO replaces the learned baseline of classic actor-critic methods with a group baseline computed from samples. For every prompt, the current policy generates a group of G completions, typically between 4 and 64, and a reward function scores each one.
Each completion then receives a group-normalized advantage: its reward minus the group mean, divided by the group standard deviation. Completions that score above the group average are reinforced, while those below it are discouraged. In the common outcome-reward setup, the same advantage is assigned to every token of a completion.
The training cycle can be summarized as a sequence of distinct steps.
| Step | What happens | Purpose |
|---|---|---|
| Sampling | Policy generates G outputs per prompt | Builds a comparison group |
| Scoring | Reward model or rule-based verifier scores outputs | Provides the learning signal |
| Normalization | Rewards are standardized within the group | Produces relative advantages |
| Policy update | Clipped surrogate loss is optimized | Limits the size of each update |
| Regularization | KL divergence to a reference model is penalized | Prevents drift from the base model |
The GRPO objective and its key parameters
The GRPO objective keeps the clipped probability ratio from proximal policy optimization. The ratio between the new and old policy probabilities is clipped to a band such as 0.8 to 1.2, so that a single batch cannot move the policy too far.
Unlike standard PPO for language models, which subtracts a KL penalty from the reward at each token, GRPO adds the KL divergence directly to the loss. The original formulation uses an unbiased, always-positive KL estimator computed against a frozen reference model, usually the supervised fine-tuned starting checkpoint.
The parameters below have the largest influence on stability and cost.
| Parameter | Typical range | Effect of increasing it |
|---|---|---|
| Group size G | 4 to 64 | Lower-variance baseline, higher sampling cost |
| Clip range epsilon | 0.1 to 0.3 | Larger and riskier policy updates |
| KL coefficient beta | 0 to 0.1 | Stronger pull toward the reference model |
| Sampling temperature | 0.7 to 1.2 | More diverse outputs within a group |
| Learning rate | About 1e-7 to 5e-6 | Faster but less stable training |
Several recent recipes set the KL coefficient to zero for reasoning tasks with reliable verifiers, accepting more drift in exchange for faster improvement. Whether this is safe depends on how well the reward resists exploitation.
How GRPO compares with PPO and related methods
GRPO sits between proximal policy optimization and simpler REINFORCE-style methods. It keeps the clipping and on-policy structure of PPO but drops the value network and generalized advantage estimation in favor of a Monte Carlo group baseline.
| Method | Baseline source | Critic network | Typical use |
|---|---|---|---|
| PPO | Learned value function with GAE | Yes | General RL and classic RLHF |
| GRPO | Mean reward of a sampled group | No | Reasoning and verifiable rewards |
| REINFORCE with baseline | Running or batch average | No | Simple policy gradient setups |
| RLOO | Mean of the other samples | No | Language model fine-tuning |
| DPO | None, offline preference pairs | No | Preference alignment without sampling |
The absence of a critic removes an entire model of similar size from GPU memory and eliminates the difficulty of training a value function on long text sequences. The price is a higher sampling cost, because many completions per prompt must be generated before each update.
When to use GRPO in practice
GRPO is most effective when rewards are cheap to compute and hard to fake. Exact-match answers, unit tests, format checks and theorem verifiers fit this pattern well, because the group comparison then reflects real differences in quality.
- Choose prompts of moderate difficulty, because a group in which every answer is correct or every answer is wrong has zero variance and yields no gradient.
- Filter or resample zero-variance groups to avoid wasting compute on prompts that teach nothing.
- Combine outcome rewards with format rewards when the model must follow a specific answer structure.
- Monitor response length and entropy, since both reveal collapse or reward hacking early.
- Use fast inference engines for sampling, as generation usually dominates total training time.
For tasks with subjective quality, such as tone or helpfulness, GRPO can still be used with a learned reward model, but the risk of reward hacking grows because the policy can exploit weaknesses of the scorer.
Known limitations and refinements of GRPO
Analyses after the original publication identified systematic biases in the GRPO loss. Averaging the loss per sequence before averaging across the group gives tokens in short answers more weight, which can encourage long incorrect responses. Dividing by the group standard deviation also overweights prompts that are very easy or very hard.
Several refinements address these issues, and the table below links each problem to its usual remedy.
| Problem | Cause | Common remedy |
|---|---|---|
| Length growth in wrong answers | Per-sequence loss normalization | Token-level loss aggregation |
| Difficulty bias | Division by group standard deviation | Center rewards without scaling |
| No gradient on solved prompts | Zero reward variance in group | Dynamic sampling of prompts |
| Entropy collapse | Symmetric clipping limits exploration | Wider upper clip bound |
| Reward hacking | Exploitable verifier or reward model | Stricter verifiers and KL control |
Variants such as Dr. GRPO and DAPO apply combinations of these fixes. The underlying idea, relative scoring within a group, remains the same, and the right variant depends on task length, reward sparsity and available compute.
Explained without jargon
Imagine a teacher who asks a student to solve the same problem eight times in different ways. The teacher checks each attempt and praises the ones that did better than the student’s own average, while discouraging the weaker ones.
GRPO trains an AI model in exactly this way. It does not need a second model to predict how good an answer should be. It simply compares answers to the same question with each other. Over many questions, the model learns which ways of thinking lead to correct results.
A typical use case
A research team wants to improve a 7-billion-parameter language model at solving school math problems. It collects about 20,000 problems with known numeric answers and writes a rule-based checker that awards 1 point for a correct final answer and 0 otherwise, plus a small bonus for using the required answer format.
For each problem, the model generates 16 solutions at temperature 1.0. Problems where all 16 are right or all are wrong are skipped, because they carry no signal. The remaining groups are normalized and used for clipped policy updates with a small KL penalty toward the starting model.
Because no critic is needed, the team fits training on half the GPUs a PPO setup would require. After several thousand steps, accuracy on a held-out test set rises noticeably, and the model begins writing longer, self-checking solutions, while the team watches response length to catch any reward exploitation.