The Paper

The Reward Model Was Hiding in the Policy

The reward model RLHF pays a full training run to fit is already implied by the policy; Direct Preference Optimization turns that identity into a single classification loss.

12 min read2026-07-11

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn · arXiv:2305.18290 (NeurIPS 2023) · 2023

Read the paper

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.¹

Before you read

This paper assumes you already picture the standard alignment pipeline. A base language model is first fine-tuned on demonstrations (SFT). Then a separate reward model is trained to score a response, fit from human comparisons of response pairs under the Bradley-Terry model, which says the probability a human prefers one response is a logistic function of the difference in their scalar rewards. Finally the policy is optimized to maximize that reward while a Kullback-Leibler (KL) penalty holds it near the reference model, so it does not chase reward off a cliff into gibberish. If the terms logits, log-probability, and KL divergence are familiar, you have what you need; the one piece of RL machinery worth naming is PPO, the policy-gradient method that usually does that last step.

Why alignment kept an RL loop it did not want

By 2022, reinforcement learning from human feedback had become the default way to make a language model helpful. InstructGPT fit a reward model from rankings, then fine-tuned the policy with reinforcement learning against it, and a 1.3B model tuned this way was preferred to raw 175B GPT-3.² The catch was the last stage. Optimizing a policy against a learned reward with PPO means sampling from the model in the loop, juggling several model copies at once, and tuning an RL algorithm that is delicate on language.³ Practitioners wanted the alignment that preferences buy without the reinforcement learning that pays for it. DPO's answer is a single move: a change of variables that makes the language model be its own reward model, so a dataset of preference pairs trains it directly with a classification loss and no sampling and no RL loop.¹

The pipeline DPO deletes

The idea of learning from comparisons rather than a hand-written reward is older than instruction tuning. Christiano and colleagues showed in 2017 that you could train agents on non-expert human preferences between pairs of trajectory segments, fitting a reward model from those comparisons and optimizing it with RL.⁴ Applied to language, the recipe has three stages. Collect pairs of responses to the same prompt and a human label of which is better. Fit a reward model r(x, y) so that preferred responses score higher, under Bradley-Terry. Then optimize the policy against that reward with PPO.²

Every failure mode DPO advertises lives in that third stage. The reward model is a frozen approximation, so a policy optimizing hard against it can find responses that score well and read badly. To keep that in check the objective adds a KL penalty toward the reference model, which then has to be tuned. And PPO itself alternates sampling from the current policy with gradient updates on a clipped surrogate objective, which is why the pipeline holds a policy, a reference, a reward model, and a value estimate in memory at once.³ The reward model is a means to an end. It exists only to define the objective the policy eventually chases.

The change of variables

That objective has a known closed-form solution. The policy that maximizes expected reward minus the KL penalty is the reference distribution tilted by the exponentiated reward:

π_r(y | x) = (1 / Z(x)) · π_ref(y | x) · exp( (1 / β) · r(x, y) )

The term Z(x) is a partition function, a sum over every possible response that normalizes the tilted distribution back into a probability. That sum is intractable for language, which is exactly why nobody solves RLHF this way and reaches for PPO instead.¹ DPO's insight is not to compute Z(x) but to route around it. Take the logarithm of both sides and solve for the reward rather than the policy:

r(x, y) = β · log( π_r(y | x) / π_ref(y | x) ) + β · log Z(x)

As we will describe next in detail, our key insight is to leverage an analytical mapping from reward functions to optimal policies, which enables us to transform a loss function over reward functions into a loss function over policies.¹

Section 4 · opening The whole paper in one sentence: the closed-form solution runs both ways, so a loss you know how to write over rewards becomes a loss over the policy itself. It does not yet claim the intractable Z(x) disappears.

Read that second equation as a definition. Any reward function corresponds to some optimal policy, and that reward can be rewritten as β times the log-ratio between that policy and the reference, plus a term that depends only on the prompt. The reward is no longer a separate network. It is a simple function of the policy you were going to train anyway.

Why the intractable term dies

The reparameterized reward still carries the intractable β · log Z(x). Here the choice of preference model earns its keep. Bradley-Terry does not depend on the rewards of two responses separately, only on their difference. When you substitute the reparameterized reward into the preference probability, both responses share the same prompt x, so both carry the identical β · log Z(x) term, and the subtraction erases it. What survives is a preference probability written entirely in quantities you can compute: the log-probabilities the policy and the reference assign to the two responses. The winning and losing completions y_w and y_l reduce to a logistic loss:

L_DPO = − E[ log σ( β log( π_θ(y_w | x) / π_ref(y_w | x) ) − β log( π_θ(y_l | x) / π_ref(y_l | x) ) ) ]

This is ordinary binary cross-entropy. It needs the log-probabilities of the preferred and dispreferred responses under the model being trained and under a frozen reference, and nothing else. No reward network, no sampling from the policy during training, no value function, no RL. The objective is differentiable, so the gradient is exact rather than estimated.¹

In contrast, DPO directly optimizes for the policy best satisfying the preferences with a simple classification objective, fitting an implicit reward model whose corresponding optimal policy can be extracted in closed form.¹

Figure 1 caption The claim is not that the reward vanishes. It is that the reward model becomes implicit in the policy, so training the classifier and training the aligned model are the same act.

What the gradient is actually doing

The loss deserves suspicion. If it only pushed up the log-probability of preferred responses and pushed down dispreferred ones, it would be a naive ratio objective, and the paper reports that such an objective degenerates. The sigmoid saves it. Differentiate the loss and a scalar weight appears in front of every update, and that weight is the sigmoid of how badly the model's own implicit reward, r̂(x, y) = β log( π_θ / π_ref ), currently ranks the pair. When the model already prefers the right response, the weight is near zero and the example barely moves the parameters. When the model has the pair backwards, the weight is large and the update is strong.

Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective.¹

Introduction The importance weight is the load-bearing detail. The authors report that removing it makes the method fail, so the sigmoid is not cosmetic.

This clears up the misreading DPO invites. It is tempting to say DPO removes reward modeling. It does not. The objective is the reward model's own training loss, and the reward still lives in the system, folded into the log-ratio of the policy against its reference.⁵ Nathan Lambert's account puts the same point structurally: DPO reaches the RLHF objective without ever training an intermediate reward network, because the log-ratio parameterization is the reward, learned by gradient descent on the preference data.⁶ The reward model was never a separate thing that had to exist. It was a view of the policy.

What the experiments show, and what they leave open

The paper tests three settings. Controlled sentiment generation on IMDb prefixes, where a sentiment classifier supplies ground-truth preferences and the reward-versus-KL frontier can be traced exactly. TL;DR summarization, judged by GPT-4 win rate against reference summaries. And single-turn dialogue on Anthropic's Helpful and Harmless data. On sentiment, DPO sits on the efficient frontier: for every level of KL divergence from the reference, it reaches higher reward than PPO. On summarization at temperature zero, DPO wins about 61% against human reference summaries where PPO reaches 57% at its best temperature, and DPO holds up better as sampling temperature rises.¹

Fig. 1 · GPT-4 win rate vs. reference summaries, temperature 0. In-distribution TL;DR and out-of-distribution CNN/DailyMail. DPO (Rafailov et al., 2023).¹

Read like a reviewer, the evidence is real but bounded. The strongest result is the frontier plot, because it measures the actual optimization objective against a reward the experimenters control rather than a proxy. The dialogue and summarization wins lean on GPT-4 as judge, which the authors defend with a human study showing GPT-4 agreement with people is typically similar to or higher than agreement between human annotators. The honest limitation is generalization. On out-of-distribution CNN/DailyMail articles the win rates fall for both methods, and the authors write plainly that whether DPO policies generalize out of distribution as well as an explicit reward model is a question their initial results only begin to answer.¹

What holds up

The derivation is exact, not an approximation of RLHF: same objective, closed-form solution.
It removes the RL loop, the reward network, and policy sampling during training, which is why practitioners adopted it fast.
On its own benchmarks it matches or beats PPO while being far simpler to run.

What to be careful about

DPO trains on a fixed offline preference set; it never samples fresh completions the way online RL does.
Out-of-distribution behavior versus an explicit reward model is left open by the authors themselves.
Later work argues the gap to a well-tuned PPO is real on hard tasks.

The three years since have argued the open question hard, and the reconstruction should not paper over it. A 2024 study tuning PPO carefully found it surpassing DPO across their benchmarks, including code competitions, and pointed to fundamental limits of DPO's offline setup.⁷ A theoretical analysis showed DPO can overfit when preferences are near deterministic, because it inherits the Bradley-Terry approximation along with the reward reparameterization.⁸ And SimPO dropped the reference model entirely, using a length-normalized average log-probability as the implicit reward, and reported gains over DPO on chat benchmarks.⁹ None of this unwinds the core identity DPO wrote down. It sharpens the verdict: the change of variables is correct and durable, and it seeded a family of direct-alignment methods, but it is a starting point for preference optimization rather than its last word.

Verdict DPO's contribution is a proof, not a benchmark number: the RLHF objective has a closed-form solution that makes the policy its own reward model, collapsing three stages into one loss. Take the derivation as settled and the empirical superiority as setting-dependent and still contested.⁷

Read it yourself

A full read adds the formal statements this reconstruction stated plainly: the theorem that the reparameterization loses no generality, the complete partition-function derivation, and the experimental tables behind the win-rate summary.

read §4 Direct Preference Optimization: the change of variables, the loss, and the gradient interpretation. This is where the whole idea lives; read it slowly.
read Appendix A.1: the derivation of the optimal policy and the reward reparameterization, the two steps the main text compresses.
skim §6 Experiments: stare at Figure 2, the reward-versus-KL frontier. It is the cleanest evidence because the reward is controlled rather than a proxy.
skim §5 Theoretical Analysis: the equivalence-class argument for why the reparameterization is fully general; skim unless you doubt the derivation.
skip §2 Related Work: the reconstruction already placed the paper against RLHF and the direct-alignment methods that followed.

One thing first-time readers miss: the sigmoid weighting is not a detail of the loss, it is the reason the method works at all. Without it the naive ratio objective degenerates.