Paper of the Day
Looped transformers turn depth into a dial
Giannou et al. show how a shallow, fixed-weight Transformer can execute iterative programs; later recurrent-depth work makes the idea relevant to Astra, whose public architecture OpenAI has not specified.
Sandbagging turns capability tests into menus
Bergemann, Koh, and Morris model an AI’s hidden preferences and capabilities as a private type, then show why a test can earn trust only when the permissions it buys are incentive-compatible.
August 2026Mamba matches the Transformer with a state that never grows
Letting a linear-time recurrence change its parameters with each token is what finally closed the gap to attention on language, and the same fixed-size state that keeps it cheap is where it still falls short of what a growing cache can recall.
FlashAttention computes exact attention without building the full matrix
What slows attention on a GPU is memory traffic more than arithmetic, and Dao and colleagues cut that traffic about ninefold by computing the attention matrix in on-chip SRAM one block at a time and writing only its result back to memory.
DPO makes the policy its own reward model and drops RLHF's RL loop
Rafailov and colleagues proved the swap targets the same KL-constrained objective as PPO-based RLHF, and no later work overturns that theorem, though the record after it locates DPO's real cost in offline practice.
Goodfellow's linear explanation got the attack right and the cause wrong
Explaining and Harnessing Adversarial Examples handed the field its one-step attack and its first practical defense, then watched the vulnerability it blamed on linear models get traced to the training data itself.
The reparameterization trick pulled randomness out of the gradient's path
Kingma and Welling's 2014 paper rewrote the encoder's random sample as a deterministic function of independent noise, making the evidence lower bound differentiable enough to train encoder and decoder together with ordinary stochastic gradient descent.
DeiT beat ViT-B/16's JFT-300M accuracy training on ImageNet alone
A modernized convolutional network on a modern recipe ties ViT at matched size, so most of what looked like a data-scale win for the transformer was the training recipe.
Frontier models kept 2017's sparse gate and rebuilt its load balancing
Shazeer and colleagues built a 4.3-billion-parameter model that beat the best published result on a one-billion-word benchmark at six percent of its compute, using a noisy gate and two balancing losses to keep 4,096 experts busy.
A controlled ablation traced most of PPO's edge to the code around its clip
The clipped objective that defines PPO is still the field's default, but a 2020 ablation holding the implementation fixed located most of its reward margin over TRPO in nine unglamorous training tricks.
CLIP's robustness came from its data, a cause the paper floated and left open
A caption-matching model matched a supervised ResNet-50 on ImageNet with none of its labels and held up under natural distribution shift, a robustness later work traced to the training data the paper had only floated.
Two fixes turned a diverging network into a 49-game Atari player
Mnih and colleagues stabilized deep Q-learning with experience replay and a frozen target network, and the reproducibility work that followed left the honest 29-of-49 result standing while exposing how much the aggregate “human-level” label was hiding.
Attention learned to align long before it was asked to explain
The 2015 model that introduced attention rebuilt its read of the source at every output word, and the alignment maps it drew look linguistic without settling what the weights explain.
Dropping bidirectionality broke BERT where dropping its sentence task did not
The recipe became the field's default encoder, and the re-examination that followed kept its bidirectional masking while discarding the next-sentence loss BERT shipped beside it.
Labelers preferred a 1.3B InstructGPT to 175B GPT-3
Ouyang and colleagues fit a reward model to human comparisons and optimized GPT-3 against it under a KL penalty, an objective whose closed form later made both the reward model and its reinforcement-learning loop optional.
Predicting the noise, not the image, is what made diffusion models work
Ho, Jain, and Abbeel reached a state-of-the-art CIFAR-10 FID of 3.17 in 2020 by dropping one weighting term from the diffusion variational bound, and every later text-to-image model inherited the objective that was left.
GANs train a loss their optimality proof leaves out
The optimum at p_g = p_data is a real theorem about function-space minimax, and the loss Algorithm 1 actually descends is the one later work shows yields an unstable, infinite-variance gradient.
Bigger models get better again past the interpolation threshold
Belkin and Nakkiran rebuilt double descent from random features to ResNets, and the after-record narrows it from a law in parameter count to a curve that depends on how you measure complexity.
word2vec's most famous demo predates word2vec
The King–Queen analogy that made the paper famous first appeared in an earlier RNN model, and the version word2vec actually ran only works with its own question words ruled out as candidates.
OpenAI's grokking paper never explained grokking
Three follow-up papers have since located the cause of the delay in three different places, and none of them agrees with the others.
July 2026Rescoring the same GPT-3 outputs turns a leap into a slope
Schaeffer, Miranda, and Koyejo's 2023 rebuttal turns Wei et al.'s sharpest curves smooth by rescoring the same outputs. It leaves open exactly what a deployment needs to know: when the metric it depends on will cross the line.
A distilled network inherits its teacher's accuracy, not its predictions
Six years after Geoffrey Hinton proposed training a small network on a teacher's softened probabilities, a controlled study traced the technique's stubborn failures to an optimization problem no one had been solving.
LoRA ties full fine-tuning where intrinsic rank predicts it should
Five years of follow-on work now show exactly where that assumption fails: on code and math tasks whose true weight updates span ten to a hundred times more directions than any practical LoRA rank reaches.
Chain-of-thought prompting tripled the largest models' math scores, and made the smallest ones worse
Below roughly 100 billion parameters the technique is a wash or a liability, and the higher a model climbs past that line, the less its written reasoning reflects the computation that actually produced its answer.
Attention alone outran recurrence, at a cost its own table had already recorded
The 2017 paper that proved recurrence and convolution were unnecessary for translation also logged the O(n²) cost that the state-space models of 2021 to 2024 built their case against.
A plain 34-layer network trained worse than the 18-layer network sitting inside it
The paper diagnosed the gap as a training failure, not overfitting, and the residual fix it proposed still has no single account the field agrees explains why it works.
A single one-dimensional problem breaks Adam's convergence proof
The fix that made Adam durable was Loshchilov and Hutter's 2019 repair to weight decay, arriving a year after Reddi et al.'s counterexample had already sent it to the worst point in its own one-dimensional search space.
Batch normalization's own paper never measured internal covariate shift
The paper credited its speedup to a change in activation statistics it never actually measured, and the team that finally measured it found batch-normalized networks carry just as much of that change as ordinary ones.
A random network hides a sparse one that trains from its first weights
A pruned subnetwork reaches full accuracy from the exact weights it started with and fails from fresh ones, and at ImageNet scale only once that starting point rewinds a few steps into training.
Chinchilla beat a model four times its size by training on four times the data
For a fixed compute budget the best split is about twenty training tokens per parameter, a rule that survived an independent replication of its weakest proof and that practice has since deliberately left behind.