Muon replaces every hidden weight update with its nearest orthogonal matrix

What momentum hands the optimizer

A hidden layer's weights are a matrix, and so is the update that a training step subtracts from them. SGD with momentum builds that update by keeping a running average of the gradients, the momentum buffer, and hands the optimizer a single matrix to apply. Every familiar optimizer then treats that matrix elementwise: Adam divides each entry by its own running scale, SGD multiplies the whole thing by one learning rate. Muon does something no elementwise rule can. It reads the update as a linear map and rebuilds it, keeping the directions the map acts along and overwriting how far it stretches each one, then applies the rebuilt matrix.¹

The idea comes with a track record before any theory. Muon was tuned on the nanoGPT speedrun, whose task is fixed and unforgiving: train a 124M-parameter GPT-2 on the FineWeb corpus until it reaches 3.28 cross-entropy on the held-out validation split, and do it as fast as possible on eight H100 GPUs. Karpathy's llm.c reproduction first hit that bar in roughly 45 minutes; the community record now clears it in under about 90 seconds.² Swapping AdamW for Muon is what set the record on 15 October 2024, a 35% cut in training time, and Muon has held the lead through every speedrun record since.¹

The matrix behind the momentum buffer

Muon's update starts exactly where SGD-momentum's does. The reference implementation keeps a momentum buffer with decay 0.95, applies a Nesterov look-ahead, and produces the update matrix G a plain momentum step would have applied.³ Nothing so far is new. What is new is refusing to read G entry by entry.

Give G a geometric body. Any matrix acts on space by stretching it along a set of orthogonal input directions, mapping each onto an orthogonal output direction, and scaling it by a non-negative number on the way. Those scale factors are the singular values, and the directions are the singular vectors; collecting them is the singular value decomposition, G = USV^⊤, where the columns of U and V are the output and input directions and the diagonal of S holds the singular values. The singular values carry all the magnitude: they say how far the update pushes along each of its own natural directions, and nothing about the coordinate axes the weights happen to be stored in.

Why a few directions swallow the update

Read the update through its singular values and a specific pathology shows up. In a trained transformer, the momentum update's singular values are wildly unequal: a handful are large and the rest are tiny. The write-up states the observation and the wager built on it in one passage.¹

Condition number is the ratio of the largest singular value to the smallest that still counts. A high one means the update is nearly concentrated in a low-dimensional subspace: written out, it is close to a sum of a few outer products, so almost every neuron's row moves along the same small set of dominant directions and hardly anywhere else. Momentum sharpens this. Averaging gradients over many steps reinforces whatever direction has been consistent and averages down the rest, so the buffer's spectrum grows spikier the longer training runs. The directions with small singular values are not noise to be discarded. Some of them carry the signal a particular neuron needs, and under a raw momentum step they are stretched so little that the neuron learns almost nothing along them while the dominant directions race ahead.

The update arrives with a fixed budget of movement handed out across its orthogonal directions, and momentum piles nearly the whole budget onto a few of them.

Setting every singular value to one

The fix is to stop letting magnitude decide which directions move. Keep the update's directions exactly as momentum found them, and set the amount of stretch along every one of them to the same value. That operation has a closed form: the semi-orthogonal matrix nearest to G, the one that minimizes the Frobenius distance to G subject to having orthonormal rows or columns. Written through the decomposition, it is simply G with its diagonal of singular values replaced by ones.¹

Dropping S is the equalization drawn in words above, made exact. Every direction now stretches by one. The dominant directions are scaled down to the same length as the rest, and the starved directions are lifted to it, so a neuron that was learning almost nothing along a rare-but-useful direction now takes a full-sized step there. The result is a matrix with orthonormal rows or columns, a semi-orthogonal matrix, which is why the operation is called orthogonalizing the update.

There is a second reason to prefer this particular matrix, and it reframes Muon as a principled step rather than a heuristic. A line of work by two of the write-up's own contributors casts common optimizers as steepest descent under different notions of size, and derives Muon's rule as the steepest-descent step when the update's size is measured in the spectral norm, its largest singular value, rather than entry by entry.⁴ Under that measure the best step of a fixed size spends its whole budget spreading movement across directions instead of concentrating it, and the optimum is exactly the semi-orthogonal matrix. The picture and the norm agree: equalizing the spectrum is what descending in spectral norm means.

Orthogonalizing without an SVD

Setting the singular values to one sounds like it needs the singular value decomposition, and computing an SVD of every weight update every step would be far too slow to matter. Muon never forms one. It reaches a near-semi-orthogonal matrix with a short sequence of matrix multiplies, the Newton–Schulz iteration, which applies the same function to the singular values as a side effect of operating on the whole matrix.

The mechanism is a fixed odd polynomial. Start from the update normalized so its singular values sit in the interval from 0 to 1. Then repeat the map φ(x) = ax + bx³ + cx⁵, applied to the matrix as X ← aX + b(XX^⊤)X + c(XX^⊤)²X. Because that expression is built only from X and its own Gram matrix, it acts on each singular value independently while leaving the singular vectors untouched: run the iteration and the singular vectors of the input are exactly the singular vectors of the output, with each singular value σ mapped to φ(σ). Push every singular value toward 1 and the matrix converges to UV^⊤, with no decomposition ever formed. The write-up states the condition for that convergence.¹

The coefficients Muon actually ships are (a, b, c) = (3.4445, −4.7750, 2.0315), and they are not the safe, slowly-converging choice the guarantee describes. They are tuned to move fast, and the price is that the polynomial overshoots. Plugging them in shows what one step does to a single singular value: a value of 0.1, a rare starved direction, is lifted to 0.34, more than tripled, while a value of 1 is pulled down to 0.70, and values in the middle are thrown briefly above 1 before later steps settle them. The map is not the gentle push toward 1 the convergence story suggests; it is an aggressive equalizer that closes the gap between large and small singular values in a few strokes.

Five iterations of this map suffice.¹ Repeated, it drives the whole spectrum into a tight band near 1: not every value pinned to exactly 1, but the largest and smallest brought close enough together that the update is near-semi-orthogonal, which is all the optimizer needs. The reference code is the map above run five times in bfloat16, transposing the matrix first when it has more rows than columns so the smaller Gram matrix is the one squared.³

1. Accumulate momentum

   Update the momentum buffer with the new gradient at decay 0.95 and take the Nesterov look-ahead, giving the update matrix G.³

2. Normalize into range

   Scale G so its singular values fall in the interval from 0 to 1, the starting condition the iteration needs.¹

3. Iterate five times

   Apply the quintic map five times as matrix multiplies, pushing every singular value toward 1 while the singular vectors stay fixed.³

4. Scale and apply

   Rescale the near-orthogonal update for the matrix's aspect ratio, then subtract it from the weights at the learning rate.³

The parameters Muon skips and the FLOPs it adds

Orthogonalizing an update only means something for a matrix, so Muon governs only the 2D hidden weights. The write-up is explicit about the boundary and about which layers to keep on a standard optimizer.¹

The embedding and output layers are the pointed exclusions. Their rows correspond to individual tokens, updated only when those tokens appear, so treating them as a single linear map to be spectrally balanced is the wrong model of what they do; the write-up finds they train better left on AdamW. Muon is therefore always paired with a second optimizer, and adopting it is a decision about the hidden layers alone.

The obvious worry is cost: five extra passes of matrix multiplication on every weight matrix every step sounds expensive. It is not, and the reason is that the iteration works on the weight matrices, whose size is set by the model dimension, while the forward and backward passes work on the whole batch of tokens. The write-up bounds the overhead as at most T·m/B, for T = 5 iteration steps, model dimension m, and batch size B in tokens. On the nanoGPT record that is 0.7% of the step's compute; scaled up to a hypothetical 405B model it is 0.5%.¹

One shape detail earns a mention because a later result explains it. The orthogonalized update is rescaled by the square root of the matrix's larger dimension. The reason is that a perfectly orthogonalized update has a root-mean-square entry of one over the square root of that larger dimension, so its typical entry size drifts with the parameter's shape.⁵ Correcting for the shape makes the update's scale uniform across the network, which is what lets a single learning rate serve every layer.

The size of the edge is a regime question

Muon replaces an elementwise update rule with a spectral one for a contained cost, and it holds a real benchmark lead. The harder question is how large the advantage is, and here the honest answer is that it depends on who measures and under what conditions. Two careful reads disagree, and the disagreement is instructive rather than a wash.

Moonshot AI's Kimi team scaled Muon to a 3B-active, 16B-total mixture-of-experts model trained on 5.7 trillion tokens, adding weight decay and the shape correction above so the tuning transfers from AdamW. Under compute-optimal training they report that Muon reaches AdamW's loss for about 52% of the training FLOPs, which they frame as roughly 2× efficiency.⁵ That is a striking number, and it is the one most often repeated. It is also a specific measurement: FLOPs to match, under their weight-decay and scale fixes, at the compute-optimal point.

An independent group at Essential AI, with no stake in Muon's origin, ran their own sweeps to four billion parameters across batch sizes from 128K to 16M tokens and read the advantage more conservatively. In their setting Muon needs 10 to 15% fewer tokens than AdamW to reach an identical loss, well short of 2×, and they locate the benefit precisely: it appears at large batch sizes, past the point where adding more tokens per step usually stops helping, and it does not shrink as the batch grows.⁶ The two results are not in contradiction because they measure different things, FLOPs-to-match at compute-optimal versus tokens-to-match across batch scales, but they set very different expectations, and only the smaller, batch-conditioned figure comes from a lab that did not build the method.

The mechanism itself is not in dispute. Every source here, including the independent one, describes Muon the same way: spectral-norm steepest descent on the hidden matrices, computed by Newton–Schulz. What is open is the payoff at the frontier. The scaling evidence in hand tops out at a 16B-total sparse model and a 4B dense one, and the write-up's own closing question asks whether the advantage survives an order of magnitude further.¹

The learning-rate and weight-transfer story sits in the same middle state. Moonshot's shape correction and weight decay let Muon inherit AdamW's learning rate and weight decay directly, and Essential AI add a telescoping protocol for tuning hyperparameters at reduced cost.⁶ Both are real progress; neither proves the recipe holds at scales no one has yet trained.

Read it yourself

The write-up is short and the reference code is shorter; together they make the update rule concrete in a way a reconstruction has to compress.¹

• read The "Definition" and "Tuning the coefficients" sections: the orthogonalization operation and the fixed-point argument behind the quintic. This is the whole method.
• read muon.py, the zeropower_via_newtonschulz5 function: five lines that turn the map into matrix multiplies. Trace one call by hand and the iteration is yours.³
• skim The Moonshot and Essential AI scaling papers: for the payoff and its regimes. Note which figure is FLOPs-to-match and which is tokens-to-match before quoting either.⁵