A random network hides a sparse one that trains from its first weights

The sparse networks nobody could train

A trained image classifier is mostly ballast. Delete the weights with the smallest magnitudes, keep going until more than ninety percent of them are gone, and the network that remains still classifies about as well as the one you started with.¹ This was settled practice by 2018: train a large model, prune it hard, fine-tune what survives, and ship a small network that runs cheaply.

The obvious next step never worked. If the pruned network is all you need, why pay to train the large one first? Take the small sparse architecture the pruning produced, initialize it fresh, and train it from scratch. In practice it learns slowly and lands at worse accuracy than the pruned-and-fine-tuned version. The sparse shape that was perfectly good after training was, for years, untrainable before it.¹

Frankle and Carbin found the missing ingredient, and it was not the shape. A pruned subnetwork can be trained from scratch to the full network's accuracy, but only if each surviving weight starts from the exact random value it was handed at the original network's birth. Keep those particular starting numbers and the sparse network trains. Draw fresh random numbers into the same holes and it does not. They named the trainable subnetwork a winning ticket, because its connections had drawn initial weights that happened to make training work.¹

Reset the survivors to their first weights

The procedure that finds a winning ticket is a small change to ordinary pruning, and the change is the whole result. Standard pruning keeps the trained values of the surviving weights. Frankle and Carbin throw those away and rewind each survivor to the value it held before any training, at initialization. Call that initial weight vector \theta_0. The steps, iterated:

1. Train the dense network

   Initialize a full network at \theta_0 and train it to completion. Record its accuracy as the target to match.¹

2. Prune the smallest weights

   Rank the trained weights by magnitude and delete the smallest fraction. What remains is a mask: a fixed pattern of which connections stay and which are set to zero for good.¹

3. Rewind the survivors to initialization

   For every connection the mask keeps, restore its value from \theta_0, not its trained value. The subnetwork now has a sparse shape and its original birth weights, nothing learned.¹

4. Train the subnetwork alone

   Train the masked network in isolation for the same schedule and compare its accuracy to the target. A winning ticket matches or beats it.¹

5. Prune again and repeat

   Removing everything in one cut finds a worse ticket than removing a little at a time. Each round prunes a further slice of the survivors, rewinds, and retrains, driving the network below twenty percent of its weights while accuracy holds.¹

Iterative pruning that keeps trained weights is ordinary. Iterative pruning that keeps the mask but restores the initialization is what uncovers a subnetwork trainable on its own. The paper calls the repeated version iterative magnitude pruning, and it is doing two jobs at once. The mask picks a sparse shape. The rewind supplies the one initialization that makes that shape learn.¹

The same holes, fresh weights, and no ticket

A procedure that finds something is only convincing if you can say what the something is. Frankle and Carbin isolate it with one control. Take a winning ticket, keep its mask exactly, and replace the rewound weights with new random draws from the same distribution. The sparse shape is untouched. Only the specific starting numbers change. The reinitialized network trains slower and reaches lower accuracy, and the gap widens as the pruning deepens. The mask alone is not the prize. The prize is the mask together with its original weights.¹

That raises a sharper question: what about the birth weights actually matters, their precise values or something coarser? Zhou and colleagues took the algorithm apart to find out. They report that the part which counts is mostly the sign of each initial weight. Keep whether each connection started positive or negative, reset the magnitudes to a constant, and the reinitialized network still trains. Lose the signs and it fails. The lottery, on this reading, is won or lost on which side of zero each surviving weight was born, more than on its exact size.²

The same group pushed the idea to its limit and found something stranger. Search for a mask over an untrained, randomly initialized network and you can find one that already classifies far above chance with no training at all, reaching 86% on MNIST and 41% on CIFAR-10. They called it a supermask. A pattern of which random weights to keep is itself a kind of learning.² The winning ticket was never only about a good place to start gradient descent. Part of the answer was baked into the choice of connections before the first step.

A rival result credits the shape, not the weights

At the same conference, a second paper read the same evidence the opposite way. Liu and colleagues studied structured pruning, which removes whole channels and filters rather than individual weights, and reported a blunt finding. Fine-tuning a pruned model gave only comparable or worse accuracy than training that same pruned architecture from fresh random weights. The inherited weights bought nothing. What mattered was the pruned shape, which they proposed to treat as the output of an architecture search.³ On the lottery ticket specifically, they reported that the winning initialization did not beat a random one in their setting.³

Both papers ran real experiments and both are right about what they measured. The disagreement is about scope. Liu and colleagues worked with structured pruning at the large learning rates standard for those methods; Frankle and Carbin used unstructured pruning of individual weights at a smaller learning rate. The winning-ticket effect turns out to be sensitive to exactly these choices. So the honest 2019 summary was not that one paper had refuted the other. It was that the effect appeared under some conditions and vanished under others, and nobody yet had the rule that told them apart.

Where resetting to iteration zero breaks

The rule was forced into the open by scale. Iterative magnitude pruning with a rewind to initialization works cleanly on small vision networks, which is all the original paper claimed. Run it on a deep residual network for ImageNet and it collapses. Reset the survivors to their iteration-zero weights and the sparse network no longer matches the full one. The famous 2019 result, taken literally at scale, is false.⁴

Frankle and colleagues repaired it by moving the rewind point. Instead of resetting each surviving weight to its value at initialization, reset it to the value it held a short way into training, between roughly a tenth of a percent and seven percent of the schedule. With that single change, iterative magnitude pruning again finds sparse subnetworks of deep networks that train to full accuracy. One is a ResNet-50 pruned to 80% sparsity that completes training and matches the original on ImageNet.⁴

Rewinding is the same idea with its one free parameter set correctly, and setting it correctly turned the lottery ticket line of work into a usable pruning recipe. Renda and colleagues compared the standard fine-tuning step against two rewinding variants, one that rewinds the weights and one that rewinds only the learning-rate schedule. Both variants beat fine-tuning. The result is a network-agnostic method that matches specialized state-of-the-art pruning on accuracy and compression.⁵ The reset that started as a scientific probe became the retraining step you would actually use.

Stability to SGD noise decides when a ticket exists

Why should the correct rewind point be initialization for MNIST and a little later for ImageNet? Frankle and colleagues answered it with a property they call stability to the noise of stochastic gradient descent. Take a network at some point in training, make two copies, and finish training each with a different random data order and augmentation. If the two finished networks land in the same low-loss region, connected by a straight path along which the loss never rises, the network was stable at that point. From there on, training reaches essentially the same place whichever random draws come next.⁶

The condition for a subnetwork to train to full accuracy is that it be stable. A small network reaches stability at initialization, so its winning tickets exist at iteration zero, exactly as the first paper found. A large network is unstable at initialization and only settles into a determined region a short way into training. Rewind such a network to iteration zero and its ticket sits in the unstable phase, where the random draws still dominate, and it cannot match the full model. Rewind instead to the point where stability begins and the ticket trains.⁶

This is the rule the two 2019 papers were missing. Init matters and you must rewind are not competing claims. They are one condition read at two scales. The original initialization carries the ticket whenever the network is already stable there, which small networks are and large ones are not. Even the practical shortcuts fall out of the same fact. If the sparse pattern is settled early, you should be able to spot it early. You and colleagues showed you can. Winning tickets, which they call early-bird tickets, can be identified in the first few epochs with cheap training, then trained the rest of the way alone, for up to 4.7 times less energy at comparable accuracy.⁷

What the next three years established

Once the stability picture was in place, the surrounding claims resolved, each in its own direction. The winning tickets are not a per-run coincidence. Morcos and colleagues generated tickets on one natural-image dataset and reused them on others, from Fashion-MNIST and SVHN through CIFAR and ImageNet to Places365. The tickets transferred, and those from larger datasets transferred better. The initialization carries an inductive bias general enough to move between tasks.⁸

The existence claim was not merely propped up by more experiments. It was proven, in a stronger form than Frankle and Carbin stated. Malach and colleagues showed that a sufficiently over-parameterized network with random weights contains a subnetwork that matches a given target network's accuracy with no training at all, only pruning. The weak hypothesis says a good subnetwork is there to be trained. The strong hypothesis, now a theorem, says a good subnetwork is already there before training, and finding it is only a matter of pruning.⁹ The supermask was the empirical shadow of this result, and the proof caught up to it.²

The one hope the program has not delivered is the practical one: finding the ticket cheaply, before training, without the full prune-and-retrain loop. Several methods try to score weights at initialization and prune once. Frankle and colleagues tested them and found the tell. Randomly shuffle the specific weights such a method prunes within each layer, or resample their initial values, and accuracy is preserved or improves. A method whose exact choices you can scramble for free is not identifying important individual weights. It is only choosing how many to cut in each layer. The winning mask, for now, still has to be earned by training.¹⁰ The literature that grew around all of this is large enough to have its own surveys.¹¹

What the claim is worth now

The paper split, under scrutiny, into a headline that needed a footnote and a core that grew sturdier. The 2019 headline, that a subnetwork trains from its literal original initialization, is true where the network is stable at the start and false where it is not. At any interesting scale the honest statement is the rewound one. That is a real correction, and it is the paper's own authors who made it. What survived is the deeper claim underneath the headline. A dense random network already contains a sparse subnetwork worth training. The choice of connections carries information on its own, and that subnetwork's starting weights are not interchangeable with fresh ones.