the-paper
The Reward Model Was Hiding in the Policy
The reward model RLHF pays a full training run to fit is already implied by the policy; Direct Preference Optimization turns that identity into a single classification loss.
The Attention Matrix You Never Write Down
Attention was never compute-bound; FlashAttention makes it exact and fast by never writing the N-by-N score matrix to GPU memory, holding the whole computation in on-chip SRAM through tiling and an online-softmax recurrence.
A causal algorithm inferred the causes of child PTSD from data no one could randomize
Run over 1,354 maltreated children in a decades-old cohort, causal discovery named four direct causes of posttraumatic stress at age 8, and the same run shows which of them a second study should expect to lose.
To Read Its Features, Give the Layer a Wider Basis
A sparse autoencoder trades the neuron for an overcomplete dictionary whose directions each carry one checkable meaning; the wins and the pushback arrive together.
AI fiction and human fiction split on narrative structure, not prose
StoryScope withholds every stylistic signal and still tells the five leading models from human authors at 93.2% macro-F1, though three plainer detectors score higher.
Mamba changes its state-space model for every token it reads
Making its parameters depend on the input buys content-based reasoning at linear cost, at the price of a fixed-size memory that cannot recall a long context the way attention can.
Muon replaces every hidden weight update with its nearest orthogonal matrix
Born in the nanoGPT speedruns, it equalizes the momentum update's singular values with five Newton–Schulz steps that never form an SVD, and its edge over AdamW depends on the regime it is measured in.
Flow Matching Trades the Score for a Velocity Field
Lipman and colleagues prove that regressing a network onto a closed-form conditional velocity field has the same gradient as the intractable objective it stands in for, which is how a state-of-the-art image generator gets trained with nothing harder than mean-squared error.
GRPO drops PPO's critic and lets the group set the baseline
Deleting the value network turns the advantage from a learned prediction into a z-score over a sampled group, the swap that made RL post-training cheap enough to become the reasoning-model default.
AlphaFold2 folds proteins by learning a geometry it can differentiate
Coevolving residues reveal which parts of a protein sit close, and AlphaFold2 turns that signal into atomic coordinates — which is also why proteins with no evolutionary cousins stay beyond it.
LoRA shrinks a fine-tuned GPT-3 checkpoint from 350GB to 35MB
By freezing the pretrained weights and training a low-rank update to the attention projections, it matches full fine-tuning where the two were compared, and costs nothing extra at inference once the update is merged back in.