GRPO drops PPO's critic and lets the group set the baseline

PPO trains two models to update one

Proximal Policy Optimization is the algorithm that carried reinforcement learning from human feedback into production, and it is an actor-critic method: two networks train together. One is the policy being improved. The other is a value network, the critic, whose only output is a scalar guess at how much reward the policy can expect from here. DeepSeekMath describes that critic as typically another model of comparable size as the policy model, one that brings a substantial memory and computational burden.¹

Group Relative Policy Optimization keeps the policy and deletes the critic. For each question it samples a whole group of answers, scores them, and uses the group's own reward statistics as the baseline the critic used to supply. That one substitution is why RL post-training became cheap enough to run at scale, and it is the algorithm DeepSeek later adopted to train DeepSeek-R1-Zero and DeepSeek-R1, obviating the need for human-labeled reasoning trajectories.²

The critic exists to lower variance

A policy gradient improves a policy by raising the log-probability of actions in proportion to how good they turned out to be. Scored on the raw return, that update is unbiased but noisy: a lucky episode shouts as loudly as a good decision. The standard fix is decades old. As Lilian Weng puts it, a widely used variation of REINFORCE is to subtract a baseline from the return to reduce the variance of the gradient estimate while keeping the bias unchanged.³

The baseline that does this is a value function, and the quantity it builds is the advantage A(s,a) = Q(s,a) - V(s): how much better an action was than what you expected before taking it. Subtract the expectation and only the surprise drives the update. This is the value network's entire job inside the objective. It is not a second opinion on the answer and it is not part of the reward. It is a variance-reducing baseline, and nothing else.

The clipped surrogate keeps a regression head

PPO wraps the policy gradient in a clipped surrogate that stops any single update from moving the policy too far. Writing \rho_t = \frac{\pi_\theta(o_t \mid q, o_{<t})}{\pi_{\theta_{\mathrm{old}}}(o_t \mid q, o_{<t})} for the ratio between the new policy and the one that generated the data, DeepSeekMath states PPO's objective as

The advantage A_t here is not free. DeepSeekMath computes it with Generalized Advantage Estimation over the rewards and a learned value function V_\psi.¹ That value function is the critic, and Schulman's original PPO makes concrete what training it costs: when the policy and value function share an architecture, the objective must add a value-function error term, a squared-error loss (V_\theta(s_t) - V^{\mathrm{targ}}_t)^2 fit to predicted returns.⁴ A regression head, trained every step to hit a moving target.

At language-model scale that head is a model of its own, and it is awkward to fit. Usually only the last token of a response carries a reward score, which, as the paper notes, complicates the training of a value function meant to be accurate at every token.¹ So the critic is expensive to hold in memory and hard to make correct. It is a strong candidate for deletion, if only something could still supply the baseline.

Swap the prediction for the group's own average

GRPO's answer is to estimate the baseline from data it already has to generate. For each question q, it samples a group of outputs \{o_1, \dots, o_G\} from the old policy, scores each with the reward model, and then, in the paper's words, normalizes those rewards by subtracting the group average and dividing by the group standard deviation.¹ The normalized reward becomes the advantage assigned to every token of that output. Put the two baselines where they can be seen at once:

Each answer is scored against how the rest of the group did on the same question. The standardization itself is not new. Karpathy, on policy gradients from pixels, recommends standardizing returns by subtracting the mean and dividing by the standard deviation before backpropagation, so the update encourages roughly half of the sampled actions and discourages the rest.⁵ What GRPO changes is where the mean and standard deviation come from. They are read off a group of completions sampled on the spot, not predicted by a network trained to know them in advance. The baseline survives; the model that produced it is gone.

The objective, with the critic gone

Assembled, GRPO's objective keeps PPO's clipped surrogate and its KL regularization and replaces only the advantage. The colored terms are the ones worth tracking as the argument moves.

One design choice is easy to miss. PPO folds its KL penalty into the per-token reward; GRPO instead adds the divergence between the trained policy and a frozen reference directly to the loss, which keeps the advantage calculation clean.¹ It estimates that divergence with an unbiased estimator credited to Schulman, guaranteed to be positive:

One model out, sixty-four samples in

The saving is real but qualitative, and the paper never puts a number on it. What leaves the training loop is one policy-sized model's worth of parameters, its optimizer state, and its activations: the critic and everything needed to keep training it.¹ DeepSeekMath's own figure states the point plainly, that GRPO foregoes the value model and estimates the baseline from group scores, significantly reducing training resources.¹

What fills the space is generation. Where PPO paid to train a second network, GRPO pays to sample a group. In DeepSeekMath's RL run that group is 64 outputs per question, each a full forward pass with no gradient, all scored by the reward model before a single policy update, at a KL coefficient of 0.04.¹ The trade is memory for sampling compute, and it lands well because inference batches cheaply while a second training model does not. It is not free of risk. A baseline read from 64 samples is noisier than one a trained value function would give, and when the group's rewards are nearly identical, that estimate gets thin.

The substitution earns its keep on the task the paper set. Trained with GRPO on roughly 144K chain-of-thought questions drawn from GSM8K and MATH, DeepSeekMath-Instruct 7B improves from 46.8% to 51.7% on MATH and from 82.9% to 88.2% on GSM8K.¹ The numbers are evidence that a critic-free baseline can carry a real RL run, not the point of the method. The point is that the run happened without ever holding a second policy-sized model in memory.

Two normalizations that bend the update

DeepSeekMath presents its two normalizations, the per-token 1/|o_i| and the division by the group's standard deviation, as neutral machinery. An independent analysis from Sea AI Lab and the National University of Singapore, published at COLM 2025 and known as Dr. GRPO, argues that both quietly bias the optimization.⁶

Dividing by |o_i| creates a response-level length bias. For a correct answer, the shorter it is, the larger the per-token update it receives; for an incorrect one, the longer it is, the more its penalty is diluted.⁶ The policy is nudged toward short right answers and long wrong ones. Dividing the centered reward by the group's standard deviation creates a question-level difficulty bias: questions whose answers are almost all right or almost all wrong have low variance, and normalizing by it inflates their weight in the update relative to questions the model is genuinely learning from.⁶ The Dr. GRPO fix is to remove both terms and keep the group mean as the baseline.⁶

Both accounts are right about the object each describes. DeepSeekMath defined an advantage and it computes the advantage it defined; the critique measures a side effect that definition has on the gradient. There is one more data point. DeepSeek's own objective changed between papers: DeepSeekMath writes GRPO with the per-token 1/|o_i| sum, while DeepSeek-R1 writes it per sequence, with a single advantage per output and no per-token average.² R1 states no reason for the change and predates the Dr. GRPO paper, so the two forms differ without the record supporting any claim that one caused the other.