Predicting the noise, not the image, is what made diffusion models work

A five-year-old model that had never made a competitive image

The forward-and-reverse diffusion idea was not new in 2020. Five years earlier, Sohl-Dickstein and colleagues had proposed exactly this shape. Destroy the structure in an image with a slow forward diffusion, then learn a reverse diffusion that restores it, for "a highly flexible and tractable generative model of the data."² The framework was tractable and it was flexible. What it was not was competitive: nobody had made it produce images that stood next to a good GAN's. A parallel line, Song and Ermon's noise-conditional score networks, reached a CIFAR-10 Inception score of 8.87 by estimating the gradient of the log data density at many noise levels and sampling with annealed Langevin dynamics.³ Promising, still short of the best adversarial models.

This paper changed one thing and won. It kept the 2015 forward and reverse chains almost untouched, and it replaced the training target. Instead of asking the network to predict the slightly cleaner image at each step, it asked the network to predict the noise that had been added. The loss became a plain squared error with a weighting term thrown away. That single substitution took CIFAR-10 from the score-based field's 25.32 FID down to 3.17, the best reported for an unconditional model at publication.¹ The rest of the paper is the derivation that explains why the substitution was available and why dropping the weight helped.

The forward process throws information away on a fixed schedule

The forward process is not learned. It is a fixed Markov chain that adds a small amount of Gaussian noise at each of T steps, shrinking the signal by a set factor as it goes.¹

Each step scales the previous image by \sqrt{1-\beta_t} and injects variance \beta_t. The paper uses a linear schedule from \beta_1 = 10^{-4} to \beta_T = 0.02 over T = 1000 steps. Run it to the end and the image is indistinguishable from a draw of \mathcal{N}(\mathbf{0}, \mathbf{I}).¹ Nothing here is a choice the model gets to make. The forward process is a definition, and the reverse process is the only part that learns.

A property worth the entire construction falls out of this definition. Because a sum of Gaussians is Gaussian, the composition of t noising steps has a closed form. You can jump straight to any noise level in one draw, without simulating the chain.¹

This closed form is what makes training affordable. Without it, every gradient step would have to march an image through the chain to reach a random t. With it, training draws a timestep uniformly and constructs \mathbf{x}_t in one operation. The \boldsymbol{\epsilon} in that equation is about to become the regression target the whole method turns on.

The step you want to learn is intractable until you condition on the clean image

The reverse process is a second Markov chain, this one with learned Gaussian transitions, that starts from pure noise and walks back to an image.¹

Training this by maximum likelihood is intractable, so the paper does what latent-variable models do and optimizes a variational bound on the negative log-likelihood. The bound decomposes over timesteps into a term at the endpoint, a denoising term at every interior step, and a final reconstruction term.¹

The obstacle sits inside L_{t-1}. The true reverse step q(\mathbf{x}_{t-1} \mid \mathbf{x}_t) is not a Gaussian anyone can write down. Condition it on the clean image \mathbf{x}_0, though, and it collapses to a closed-form Gaussian with a known mean and variance.¹ That is the move that makes the bound computable, and it is available only because training has \mathbf{x}_0 in hand.

The paper fixes the reverse variance to \tilde\beta_t rather than learning it, so only the mean is left to match. With both sides Gaussian, the KL term reduces to a scaled squared distance between the two means.¹

The problem is now a regression. The only open question is what, exactly, the network should regress onto.

Predict the noise, then drop the weight

Two parameterizations of that regression are available, and they are a genuine fork. Predicting the mean asks the network to output where the slightly cleaner image sits. Predicting the noise asks it to output what was added. They are algebraically interchangeable, because the closed-form marginal already ties \mathbf{x}_t, \mathbf{x}_0, and \boldsymbol{\epsilon} together. The paper substitutes that relation into \tilde{\boldsymbol{\mu}}_t and finds the mean the network must produce.¹

Take a single timestep to see what this does to the loss. At that t, matching the mean \tilde{\boldsymbol{\mu}}_t becomes matching the noise once the substitution is made. The network sees the noised image \mathbf{x}_t = \sqrt{\bar\alpha_t}\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\boldsymbol{\epsilon}. It is scored on how close its \boldsymbol{\epsilon}_\theta lands to the exact \boldsymbol{\epsilon} that produced it. Do that for every t, and the interior loss is the noise-prediction term below. It carries a coefficient that depends on the step.¹

The decisive move is to throw that coefficient away and train on the bare squared error. This is not a derivation. It is a choice, and the paper defends it with a result rather than an identity. The dropped coefficient is largest at small t, where the image is nearly clean and denoising is easy. Keeping it would spend most of the training signal on the easiest steps. Setting it to one up-weights the hard, heavily-noised steps, and the authors report that this trains more stably and yields better samples.¹

What that objective buys is a training loop with no adversary, no reconstruction decoder, and no per-step targets to precompute. Sample a clean image, sample a timestep, sample noise, and take a gradient step on the squared error. Sampling reverses the same network: start from noise and walk the chain back, subtracting the network's noise estimate at each step and adding a controlled amount of fresh noise until the last step.

The parameterization is not a matter of taste, and the paper's own ablation settles it. Predicting the mean under the true weighted bound reaches an FID of 13.51. The same architecture, switched to noise prediction and trained on the unweighted L_{\mathrm{simple}}, reaches 3.17. Trying to also learn the reverse variances, rather than fixing them, was unstable.¹

Strong samples, a likelihood that lost, and a cost left unnamed

The headline number is CIFAR-10. The L_{\mathrm{simple}} model scores an FID of 3.17 and an Inception score of 9.46, the best FID among the unconditional entries the paper tabulates and ahead of the score-based and adversarial baselines it lists.¹

At 256x256 the paper shows LSUN Church samples at an FID of 7.89 and LSUN Bedroom at 4.90, with the scores printed in the figure captions rather than a comparison table.¹ The abstract claims parity here, "sample quality similar to ProgressiveGAN," and hedges to a 2018 adversarial model. One of those Church samples reproduces a stock-photo watermark from the training set intact, a visible instance of the training-data memorization these models can exhibit at this resolution.

The number the abstract does not lead with is log-likelihood, and the reason is that the winning model loses on it. The L_{\mathrm{simple}} model reaches a negative log-likelihood bound of at most 3.75 bits per dimension. The paper's own variant, trained on the true variational bound with a fixed isotropic reverse variance, does better on likelihood at 3.70 and worse on samples. The objective that maximizes image quality is not the objective that maximizes likelihood. The paper concedes the codelengths "are not competitive with other types of likelihood-based generative models."¹

The second gap is one the paper never names as a limitation. Sampling runs the reverse chain for all T = 1000 steps, one network evaluation each, and the only stated reason for that count is to match the neural-network evaluations of prior work.¹ A thousand sequential forward passes to draw one image is treated as a setting, not a cost. It is the sharpest distance between how the paper framed its method and how the field would come to see it.

A training recipe the whole field kept

Read as a reviewer, the paper's real contribution is narrower than its abstract and more durable than its own numbers. The forward-and-reverse diffusion framework is Sohl-Dickstein 2015, and the score-matching and Langevin half of the "novel connection" is Song and Ermon 2019.²³ What is new here is the \boldsymbol{\epsilon}-parameterization and the unweighted L_{\mathrm{simple}} objective, the two changes that turned a tractable-but-uncompetitive idea into the best unconditional image model of its moment. That is a training recipe, not a model family. The recipe is what the field kept.

The two soft spots were both closed within a year, and by follow-ups that read directly against them. DDIM keeps the identical training objective but replaces the sampler with a non-Markovian process, reaching comparable quality 10 to 50 times faster, which reframes the 1000-step chain as one point on a speed-quality curve rather than a fixed price.⁴ Nichol and Dhariwal answer the likelihood gap on its own terms. Learn the reverse variances the paper held fixed, add a cosine schedule and a hybrid loss, and DDPMs reach competitive log-likelihoods. The same changes cut sampling to an order of magnitude fewer forward passes at negligible quality cost.⁵ Song and colleagues then showed DDPM and the score networks are two discretizations of one stochastic differential equation. Its probability-flow ODE computes exact likelihoods and reaches an FID of 2.20 on CIFAR-10, evidence of how much headroom the family still held.⁶

Even the hedged LSUN claim understated the method. Within two years Dhariwal and Nichol added classifier guidance and architecture search, and had diffusion models beating the strongest GANs on ImageNet. The abstract's "similar to ProgressiveGAN" was a floor, not a ceiling.⁷ The plain unconditional objective extended to controllable generation without a separate classifier through classifier-free guidance,⁸ and moving the same denoising process into a pretrained autoencoder's latent space made it cheap enough for public text-to-image systems.⁹ Every one of those inherited L_{\mathrm{simple}} as written.