Frontier models kept 2017's sparse gate and rebuilt its load balancing

The bill conditional computation promised to cut

A dense network pays for every parameter on every example. Doubling the parameter count to hold more of a training corpus roughly doubles the arithmetic each token costs, in training and again at inference. That coupling is the wall the paper is built against: capacity and per-example compute rise together, so a bigger model is a slower one.¹

Conditional computation is the idea of breaking the coupling by running only part of the network per example. It had been proposed years earlier, and it had not delivered. The abstract is blunt about why: there were significant algorithmic and performance challenges in practice, not in theory.¹ The paper's answer is a single layer. A trainable gating network reads each input, picks a handful out of thousands of small feed-forward experts, runs only those, and adds their outputs back together. The rest of the experts, for that input, do no work.

A layer that mostly stays switched off

The layer holds n experts, each its own small feed-forward network with its own parameters, and one gating network G. On input x the gate emits a vector G(x) of n weights, and the layer returns the weighted sum of the experts it points to.²

Separate expert networks and a gate that learns which expert handles which cases is an old design, introduced by Jacobs, Jordan, Nowlan and Hinton in 1991, and Hinton is an author here too.³ What 2017 adds is the word sparse. In the original the gate spread soft weight across all experts, so all of them ran. Here the gate is forced to zero out all but a few weights, which is why the parameter count can grow without the arithmetic growing with it. The saving is real only if a zero weight means the expert is genuinely never run, so the gate has to produce exact zeros, not small numbers.

Why the gate adds noise before it ranks

The simplest trainable gate is a softmax over a learned linear projection of the input. It is differentiable and it spreads weight across every expert, which is exactly the density the layer cannot afford.²

Sparsity comes from keeping only the k largest entries before the softmax and sending the rest to negative infinity, so their softmax weight is exactly zero. That is the \mathrm{KeepTopK} step, and with it the gate over n experts fires only k of them, with k as small as two.²

Left there, the top-k cut has a problem the paper has to fix before the gate can be trained: it is a hard ranking. If the same few experts start out slightly ahead, they get chosen, they get gradients, they improve, and they stay ahead. Nothing pushes a good but currently unranked expert into the top k. The noise term is what breaks that lock. Before ranking, the gate adds Gaussian noise to each expert's score, scaled per expert by a second learned projection passed through softplus. This is the equation the whole gate turns on.

Because the amount of noise is itself learned, the gate can quiet the wobble for an expert it is sure about and raise it for one it wants to keep in contention. The noise does a second job, the one the load loss depends on: it makes the count of examples reaching an expert a smooth, differentiable quantity the network can be trained to balance.²

Balance means two different things at once

Left to the main objective alone, the gate collapses. A few experts win early, take most of the traffic, and the rest wither. The paper stops this with balancing losses added to the training objective, and it runs two of them because there are two distinct things that can go out of balance.²

The first is importance: the total soft weight the gate assigns to each expert across a batch. Summing G(x) over the batch gives a per-expert importance, and penalizing the squared coefficient of variation of those totals pushes the gate to spread its weight evenly.²

Equal importance is not the same as equal work. Two experts can carry the same total gate weight while one is chosen by many examples at low weight and the other by a few at high weight. On the hardware, what costs money is how many examples land on each expert, and importance leaves that free. So the second loss balances load: the count of examples routed to each expert. That count is an integer, and an integer has no useful gradient, which is where the noise from the previous section pays off. The paper replaces the hard count with the probability each example would have reached the expert under a fresh noise draw, a smooth quantity it can sum and penalize.²

The smooth count is the paper's way around a problem Bengio, Léonard and Courville had catalogued in 2013: getting a gradient through a hard, on-or-off gating unit. Their survey studied estimators such as the straight-through gradient for exactly this situation.⁴ The 2017 layer sidesteps the hard estimator by defining a soft probability it can differentiate directly.

The batch problem sparse routing creates

Sparsity does not make the compute vanish. It moves the cost, and one place it lands is batch size. Modern accelerators are efficient only on large batches, and routing splits the batch across experts. If k of n experts fire on a batch of b examples, each expert sees about kb/n of them. With thousands of experts, that per-expert batch collapses toward nothing, and each expert runs on a tiny slice at ruinous efficiency.²

The paper's fix is to make the batch each expert sees larger than the batch any one device holds. Mixing data and model parallelism across d devices, and keeping one shared copy of each expert that every device sends its routed examples to, raises the per-expert batch to about kbd/n. Applying the layer convolutionally across the timesteps of the recurrent stack enlarges it again, since every timestep contributes examples.² The other cost is communication: every routed example crosses the network to its expert's device and back, and the paper keeps that affordable by holding the expected traffic per device fixed as it adds experts.

It is not the route the field later took. The later large-scale systems cap each expert at a fixed capacity and drop the examples that overflow rather than grow every expert's batch. The Switch Transformer caps each expert with a capacity factor and drops the tokens that overflow it, typically under one percent.⁵ GShard sets a per-expert capacity too and routes each token to at most two experts, dispatching the second only with probability proportional to its gate weight.⁶ Same throughput problem, different answer.

Capacity at six percent of the compute

The claim to test is that this buys capacity almost for free. The cleanest reading is the controlled comparison on the one-billion-word language benchmark, where per-example compute is held fixed and only capacity moves. The paper reports that its fastest sparse model beats the best previously published result while using about six percent of that baseline's computation, controlling for training epochs.² Held instead at the baseline's compute, the sparse model does far more with it.

The left plot is the capacity claim in one picture: with per-example compute pinned, each added expert lowers perplexity, and the curve keeps falling out to billions of parameters. The right plot reads it the other way, showing that the sparse models reach any given perplexity far down and to the left of where the dense baselines reach it. Three rows from that curve fix the comparison.

At about a sixteenth of the baseline's per-step compute the sparse model already edges ahead, 34.1 against 34.7. Given roughly the baseline's compute budget, it reaches 28.0. That is the concrete version of the capacity claim, and it is a good deal smaller than the abstract's headline. The greater than 1000x in the abstract is a ceiling on capacity relative to a dense network of comparable per-example compute, not a single measured row.¹ The 1B-word models above hold about 4.3 billion parameters against the baseline's 151 million, roughly a thirtyfold gap. The parameter count only reaches the thousandfold scale in a separate model built on a hundred-billion-word corpus, with 131,072 experts and 137 billion parameters, at 28.9 to 29.2 perplexity.²

The gains are not confined to language modeling. On WMT'14 English-to-French translation, a sparse model with 2,048 experts scored 40.56 BLEU against a strong GNMT baseline's 39.22, a gain of 1.34, with lower perplexity as well.² A single multilingual model beat the monolingual baselines on 8 of 12 language pairs.²

Which parts of the 2017 design the field kept

The sparse top-k gate is now standard in large models. The paper stated a premise that later work directly contradicted: that routing to more than one expert was needed to give the router non-trivial gradients. The Switch Transformer routes each token to a single expert, reports that this preserves quality and lowers routing and communication cost, and scales to 1.571 trillion parameters with a four-fold pre-training speedup over a tuned dense baseline.⁵ The two-loss balancing did not survive either. Switch collapses importance and load into one differentiable term, and GShard uses a single auxiliary loss of the same family.⁵⁶

Later work questions whether a balancing loss belongs in the objective at all. Expert-choice routing inverts the direction: instead of each token picking its top experts, each expert picks its top tokens, so every expert fills a fixed bucket and load is even by construction, with no auxiliary loss. It reports more than a two-fold improvement in training convergence at equal compute against both top-1 and top-2 routing.⁷ DeepSeek's loss-free balancing keeps token-choice routing but drops the loss for a per-expert bias added to the routing scores before the top-k cut and nudged up or down by each expert's recent load. Its authors argue the balancing loss is not merely unnecessary but harmful, injecting non-negligible interference gradients that impair the model.⁸ That strategy is in production: DeepSeek-V3, 671 billion parameters with 37 billion active per token, states in its abstract that it uses an auxiliary-loss-free balancing strategy at scale.⁹

The 2017 losses do keep a foothold where routing stayed token-choice. DeepSeekMoE, which segments experts more finely and reserves a few always-active shared experts, still trains with expert-level and device-level auxiliary balancing losses of the collapsed kind.¹⁰ The record does not resolve into one verdict on balancing, and it should not be forced to. Switch and GShard kept a loss and shrank it to one term; expert-choice removed the need for it by construction; loss-free balancing removed it and calls it a liability.