Group relative policy optimization

Also known as: GRPO

On this page 7

Group relative policy optimization (GRPO) is a policy gradient algorithm in reinforcement learning that estimates the advantage of each sampled output by comparing it with other outputs generated for the same prompt, instead of relying on a learned value function. It was introduced in 2024 with the DeepSeekMath work and became widely known as the training method behind the reasoning model DeepSeek-R1.

GRPO is a simplification of proximal policy optimization designed for fine-tuning large language models. By removing the critic network, it cuts memory use and engineering complexity, which makes it a common choice for reinforcement learning with verifiable rewards, such as math answers or code that must pass tests.

How group relative policy optimization computes advantages

GRPO replaces the learned baseline of classic actor-critic methods with a group baseline computed from samples. For every prompt, the current policy generates a group of G completions, typically between 4 and 64, and a reward function scores each one.

Each completion then receives a group-normalized advantage: its reward minus the group mean, divided by the group standard deviation. Completions that score above the group average are reinforced, while those below it are discouraged. In the common outcome-reward setup, the same advantage is assigned to every token of a completion.

The training cycle can be summarized as a sequence of distinct steps.

StepWhat happensPurpose
SamplingPolicy generates G outputs per promptBuilds a comparison group
ScoringReward model or rule-based verifier scores outputsProvides the learning signal
NormalizationRewards are standardized within the groupProduces relative advantages
Policy updateClipped surrogate loss is optimizedLimits the size of each update
RegularizationKL divergence to a reference model is penalizedPrevents drift from the base model
Steps of one GRPO training iteration

The GRPO objective and its key parameters

The GRPO objective keeps the clipped probability ratio from proximal policy optimization. The ratio between the new and old policy probabilities is clipped to a band such as 0.8 to 1.2, so that a single batch cannot move the policy too far.

Unlike standard PPO for language models, which subtracts a KL penalty from the reward at each token, GRPO adds the KL divergence directly to the loss. The original formulation uses an unbiased, always-positive KL estimator computed against a frozen reference model, usually the supervised fine-tuned starting checkpoint.

The parameters below have the largest influence on stability and cost.

ParameterTypical rangeEffect of increasing it
Group size G4 to 64Lower-variance baseline, higher sampling cost
Clip range epsilon0.1 to 0.3Larger and riskier policy updates
KL coefficient beta0 to 0.1Stronger pull toward the reference model
Sampling temperature0.7 to 1.2More diverse outputs within a group
Learning rateAbout 1e-7 to 5e-6Faster but less stable training
Main GRPO hyperparameters

Several recent recipes set the KL coefficient to zero for reasoning tasks with reliable verifiers, accepting more drift in exchange for faster improvement. Whether this is safe depends on how well the reward resists exploitation.

GRPO sits between proximal policy optimization and simpler REINFORCE-style methods. It keeps the clipping and on-policy structure of PPO but drops the value network and generalized advantage estimation in favor of a Monte Carlo group baseline.

MethodBaseline sourceCritic networkTypical use
PPOLearned value function with GAEYesGeneral RL and classic RLHF
GRPOMean reward of a sampled groupNoReasoning and verifiable rewards
REINFORCE with baselineRunning or batch averageNoSimple policy gradient setups
RLOOMean of the other samplesNoLanguage model fine-tuning
DPONone, offline preference pairsNoPreference alignment without sampling
GRPO compared with neighboring policy optimization methods

The absence of a critic removes an entire model of similar size from GPU memory and eliminates the difficulty of training a value function on long text sequences. The price is a higher sampling cost, because many completions per prompt must be generated before each update.

When to use GRPO in practice

GRPO is most effective when rewards are cheap to compute and hard to fake. Exact-match answers, unit tests, format checks and theorem verifiers fit this pattern well, because the group comparison then reflects real differences in quality.

  • Choose prompts of moderate difficulty, because a group in which every answer is correct or every answer is wrong has zero variance and yields no gradient.
  • Filter or resample zero-variance groups to avoid wasting compute on prompts that teach nothing.
  • Combine outcome rewards with format rewards when the model must follow a specific answer structure.
  • Monitor response length and entropy, since both reveal collapse or reward hacking early.
  • Use fast inference engines for sampling, as generation usually dominates total training time.

For tasks with subjective quality, such as tone or helpfulness, GRPO can still be used with a learned reward model, but the risk of reward hacking grows because the policy can exploit weaknesses of the scorer.

Known limitations and refinements of GRPO

Analyses after the original publication identified systematic biases in the GRPO loss. Averaging the loss per sequence before averaging across the group gives tokens in short answers more weight, which can encourage long incorrect responses. Dividing by the group standard deviation also overweights prompts that are very easy or very hard.

Several refinements address these issues, and the table below links each problem to its usual remedy.

ProblemCauseCommon remedy
Length growth in wrong answersPer-sequence loss normalizationToken-level loss aggregation
Difficulty biasDivision by group standard deviationCenter rewards without scaling
No gradient on solved promptsZero reward variance in groupDynamic sampling of prompts
Entropy collapseSymmetric clipping limits explorationWider upper clip bound
Reward hackingExploitable verifier or reward modelStricter verifiers and KL control
GRPO weaknesses and common remedies

Variants such as Dr. GRPO and DAPO apply combinations of these fixes. The underlying idea, relative scoring within a group, remains the same, and the right variant depends on task length, reward sparsity and available compute.

Explained without jargon

Imagine a teacher who asks a student to solve the same problem eight times in different ways. The teacher checks each attempt and praises the ones that did better than the student’s own average, while discouraging the weaker ones.

GRPO trains an AI model in exactly this way. It does not need a second model to predict how good an answer should be. It simply compares answers to the same question with each other. Over many questions, the model learns which ways of thinking lead to correct results.

A typical use case

A research team wants to improve a 7-billion-parameter language model at solving school math problems. It collects about 20,000 problems with known numeric answers and writes a rule-based checker that awards 1 point for a correct final answer and 0 otherwise, plus a small bonus for using the required answer format.

For each problem, the model generates 16 solutions at temperature 1.0. Problems where all 16 are right or all are wrong are skipped, because they carry no signal. The remaining groups are normalized and used for clipped policy updates with a small KL penalty toward the starting model.

Because no critic is needed, the team fits training on half the GPUs a PPO setup would require. After several thousand steps, accuracy on a held-out test set rises noticeably, and the model begins writing longer, self-checking solutions, while the team watches response length to catch any reward exploitation.