GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity

Jul 16, 2026

Upload by Jiven

Three popular training methods for verifiable language-model reasoning—Group Relative Policy Optimization (GRPO), Dr. GRPO, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)—all act through a single quantity: the standard deviation of group rewards produced by sampling multiple responses for one prompt. Disagreement among verifier marks drives the learning signal, vanishing for unanimous outcomes and maximizing near even splits. For binary rewards, the paper proves the group-standard-deviation identity: the update magnitude equals the gradient length scaled by σ, giving a precise account of learning strength, difficulty bias, and sampling behavior, confirmed on Big-Math and controlled experiments.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

Introduction

Reinforcement learning with verifiable rewards

The lens: group reward standard deviation σ

Finite-group analysis and the identity

arXiv :2607 .00152v1 [ cs .LG] 30 Jun 2026 GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity Yong Yi Bay* Kathleen A. Yearick* PhD, University of Illinois at Urbana-Champaign AB S T RAC T Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt’s sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly. Keywords GRPO · reward normalization · silent groups · difficulty bias · group size · dynamic sampling · RLVR · LLM reasoning 1 Introduction Teaching a language model to reason starts with a step that looks wasteful: the same prompt is answered many times over, on purpose. This happens during reinforcement learning, not when a user chats with the model. The model produces a group of candidate answers, an external verifier marks each one as correct or incorrect, and the optimizer updates the model so that rewarded answer paths become more likely. The repeated answers are not redundant outputs for a user; they are measurements of the model’s current uncertainty on that prompt. A simple example captures the mechanism. Suppose a model attempts the same math problem eight times. If all eight attempts are wrong, there is no successful attempt to imitate. If all eight are right, there is no failed attempt to move away from. The useful training case is mixed: some attempts are right and some are wrong. Only then can the training rule compare the two sides. In this sense, a prompt teaches through its within-group disagreement. Group Relative Policy Optimization [GRPO; 1, 2], the workhorse of current verifiable reasoning training, is built around this comparison. GRPO operates in the reinforcement learning with verifiable rewards setting * Equal contribution. Correspondence: {yongyibay, [kallie.a.yearick}@gmail.com](kallie.a.yearick}@gmail.com). (RLVR), where an automatic checker returns a reward, usually 1 for a correct final answer and 0 for an incorrect one. The model supplies the candidate answers; the verifier supplies the rewards; GRPO supplies the rule that converts those rewards into advantages; the optimizer changes the model parameters. Figure 1: The training-time loop studied in this paper. The trainer samples one prompt many times to compare its correct and incorrect attempts, and the group reward standard deviation σ measures whether they disagree. The mean subtraction inside GRPO needs little controversy: subtracting any action-independent baseline preserves the policy gradient while reducing variance [3] . The contested step is the next one, division by the group standard deviation. It is often dismissed as a normalization detail, yet Liu et al. [4] identify it as the source of a question-level

FAQs

Why does this training approach sample multiple answers for the same prompt?

It measures the model’s uncertainty by comparing correct versus incorrect candidates produced within a group. The verifier assigns rewards (typically 1 or 0), enabling the optimizer to update using within-group disagreement.

What is the key quantity that unifies GRPO, Dr. GRPO, and DAPO?

The group reward standard deviation σ. GRPO divides by σ, Dr. GRPO omits that division, and DAPO discards groups where σ equals zero.

How does the paper relate σ to the strength and direction of the policy update?

For binary rewards, the update decomposes into a direction term comparing mean scores of correct versus incorrect rollouts and a multiplier σ that vanishes for unanimous groups and peaks for balanced splits. This yields the group-standard-deviation identity tying σ to the gradient magnitude.

Jul 16, 2026

Upload by Jiven