To Read Its Features, Give the Layer a Wider Basis

Why a neuron will not hold still

Open a trained language model, pick a neuron in the residual stream, and read off the contexts where it fires. You will usually find a mess: it activates on several concepts that have nothing to do with each other, and no single English sentence describes when it turns on. The paper's word for this is polysemanticity, and it is the reason you cannot interpret a network by reading its neurons one at a time.¹

Cunningham and colleagues take a specific hypothesis about why this happens and turn it into a recipe for undoing it. If a layer stores more distinct features than it has neurons by spreading them across an overcomplete set of directions, then the features are still in there; they are just not aligned with the coordinate axes you were reading. The move is to learn those directions directly, by training a wide, sparse autoencoder to reconstruct the layer's activations. What comes out is a dictionary of directions, each of which is far closer to meaning one thing than any neuron was.¹

Packing more features than there are dimensions

A layer can carry more distinct features than it has dimensions. Take the smallest case that still shows the effect. Suppose you have a two-dimensional activation space and you want to store feature directions in it, one unit vector per feature. If you insist that the directions be orthogonal, so that activating one feature leaves the others reading exactly zero, you can store two.

Now relax the demand. Lay down five unit vectors spread evenly around the circle. Any two of them are close to, but not quite, orthogonal. When one feature is active, each of the others reads a small nonzero value: interference. Without sparsity this cross-talk would be fatal, every feature smeared across every direction at once. The saving assumption is that the features are sparse. At any given token, almost all of them are off, so at most one or two of your five directions are genuinely active and the rest contribute only faint noise that a downstream nonlinearity can threshold away. Elhage and colleagues make this precise in small ReLU networks trained on synthetic sparse features, where the packing, the interference, and a sharp phase change between regimes can all be inspected directly.²

In a space of dimension n you can fit only n orthogonal vectors, yet you can fit exponentially many that are merely almost orthogonal, a consequence of the Johnson–Lindenstrauss lemma. A model that has learned to represent thousands of rare features in a residual stream of a few hundred dimensions is exploiting exactly this room.²

No axis is privileged in the residual stream

There is a second reason the neuron fails as a unit, and it is sharper than superposition. Asking whether a neuron is interpretable only makes sense when the basis it belongs to is special. An activation function such as ReLU acts coordinate by coordinate, which singles out the standard axes and gives them meaning the model can use. Elhage and colleagues call such a basis privileged.²

The residual stream has no such privilege. It is a sum of contributions written by many layers, and nothing in the architecture rotates it to favor the coordinate axes. Its directions are set by training, not by an activation function, so a feature the model uses has no reason to line up with any particular neuron. That is why the paper hunts for directions rather than neurons, and why the authors find the residual-stream basis no more interpretable than a random one.¹

Sparsity is what makes the packing pay

Superposition is not free. The interference between non-orthogonal directions is a cost the model accepts only when packing more features outweighs it, and the paper is explicit that the trade closes only when the features are sparse—the condition on which everything else rests.

Read that condition backward and it hands you an algorithm. If each activation vector is a sparse sum of a fixed set of feature directions, then recovering the features means finding a set of directions such that every activation reconstructs from a sparse combination of them. That is the classical problem of sparse dictionary learning, going back to Olshausen and Field in 1997, and the authors point at it by name. The only question left is how to fit it at the scale of a language model.¹

An autoencoder built to run overcomplete

The fit is a one-hidden-layer autoencoder, and its whole design is bent toward producing a sparse, overcomplete code. Cache the activations you want to decompose: the paper mainly studies residual streams in Pythia-70M and Pythia-410M, of width d_in = 512 and 1024. The hidden layer is deliberately wider than the input, with size d_hid = R · d_in, where the expansion ratio R sets how many dictionary directions you allow.¹

The encoder lifts an activation x into that wider space and rectifies it, c = ReLU(Mx + b), so the code c is nonnegative and, under pressure, mostly zero. The decoder reads the same weights back the other way, x̂ = M^Tc = Σ c_i f_i: the reconstruction is a sum of dictionary directions f_i, each scaled by its code entry. The rows of M are the dictionary, the encoder and decoder share weights by transposition, and each row is normalized to unit length. That detail matters: without it the model could shrink the sparsity penalty by inflating the feature vectors instead of using fewer of them.¹

The objective is two terms. Reconstruct the activation, and pay for every unit of code you switch on: L(x) = ‖x − x̂‖²₂ + α‖c‖₁. The squared error is reconstruction; the L1 term is the sparsity price, weighted by a single coefficient α.

The whole apparatus is cheap. The authors train with Adam at learning rate 1e-3 on five to fifty million cached activation vectors for one to three epochs, a run that finishes in under an hour on a single A40. The headline settings for the residual stream use α = 0.00086 and R = 2, chosen from sweeps over R in {0.5, 1, 2, 4, 8}.¹

Turning "interpretable" into a number you can check

Interpretable is a claim, not a property you can assert, so the paper measures it. The metric borrows an automated procedure: collect text where a feature fires, ask a language model to write a description of when it fires, then ask a second model to predict the feature's activation on held-out text using only that description. The correlation between predicted and actual activation is the feature's interpretability score. In the paper's pipeline the descriptions come from GPT-4 over the top fragments, and the simulation runs on GPT-3.5 across a mix of high-activating and random fragments.¹

Scored this way, the dictionary directions beat the alternatives. The authors compare against the default neuron basis, random directions, principal component analysis, and independent component analysis, and the learned features come out more interpretable than all four. The scores are noisy: the first five features of one layer-1 residual dictionary score 0.33, −0.11, 0.55, 0.57, and 0.19. One of them is anticorrelated with its own description. The advantage is real but graded, shrinking with depth to match ICA by the middle layers and nearly vanish in the last one.¹

The metric is only as trustworthy as its judges, and the authors note it struggles with next- or previous-token features. A cleaner test is causal. On the indirect object identification task, they patch a small set of dictionary features into the residual stream toward a counterfactual answer. The features achieve the same edit, by KL divergence from the target, in fewer patches than a comparable component decomposition, and the effect vanishes if the dictionary is trained without any sparsity penalty.¹

Anthropic's parallel work supplies the existence proof the metric cannot. Training sparse autoencoders on a one-layer model's MLP, Bricken and colleagues recover features that fire on things no neuron tracks, such as Hebrew script, and show the recovery is causal: activating a base64 feature makes the model emit base64. They also give the best evidence that the features are the model's own rather than the fitting method's. A feature in one training run has a median activation correlation of 0.72 with its nearest match in an independently trained run, against 0.46 for the nearest neurons.³

Where the dictionary stops working

The first limit is that reconstruction is lossy. The dictionaries never drive the reconstruction error to zero, which means they do not capture everything a layer holds. When the authors replace layer-2 activations in Pythia-70M with the SAE's reconstruction, perplexity on the Pile rises from 25 to 40. That fifteen-point tax is the fidelity you give up to read the layer.¹

Reconstruction and interpretability also pull against each other. A larger α buys a sparser, more readable code and a worse reconstruction; a smaller one does the reverse. Worse for anyone hoping to declare victory, the authors find no knee in the curve that trades the two off, which they read as evidence that there is no single correct way to decompose an activation space into a sparse basis. The method returns a decomposition, not the decomposition.¹

The pipeline is also uneven across sites in the network. On residual streams the authors can push to at least four times overcomplete before features start dropping out, but inside the MLP their dictionaries learn many dead features that never fire, so much so that a dictionary with twice as many features as neurons can have fewer than half of them alive. The Anthropic work makes the MLP tractable, but only by adding machinery this paper lacks: it periodically resamples dead features onto data the autoencoder reconstructs poorly, reviving wasted capacity.³

Finally, the dictionary you get depends on how big you make it, which complicates the clean story that one direction equals one concept. As Bricken and colleagues scale the number of features, a single feature for base64 splits into three and then into many, each a finer shard of the same concept.

Whether the features are the model's own

The paper's own admission that there is no single correct sparse decomposition is the crack later work has widened from three directions at once. Read together, they say the anchor proved something real and narrower than its title promises.

The first pressure is on the objective. The L1 penalty is a blunt instrument: it pushes codes toward zero but also shrinks the codes that survive, and tuning α has no natural stopping point, the smoothness the anchor observed. Gao and colleagues at OpenAI replace it with a k-sparse constraint that fixes the number of active features directly, report that this cleans up the reconstruction-sparsity frontier and nearly eliminates dead latents, and find clean scaling laws, training a sixteen-million-latent autoencoder on GPT-4 activations.⁴ It is not the only fix. The same shrinkage bias, the L1 penalty systematically underestimating the activations it lets through, is the explicit target of Gated sparse autoencoders, which make only a gate deciding which features fire pay the L1 price while a separate branch sets magnitude unpenalized,⁵ and of JumpReLU autoencoders, which swap the ReLU for a thresholded activation and train the sparsity count directly rather than through an L1 proxy.⁶ The anchor's L1 recipe is already one revision behind, on more than one front.

The second pressure is on usefulness. If the features are the model's real variables, they should earn their keep on downstream tasks. Kantamneni and colleagues test that on sparse probing and do not find it.

The third pressure is on the clean-feature story itself. Chanin and colleagues name a failure mode, feature absorption, in which a feature that looks monosemantic quietly fails to fire on cases that belong to it, because a more specific feature has absorbed them. Optimizing for sparsity over concepts that are really hierarchical produces features that are neither clean nor complete, which is feature splitting seen from its damaging side.⁸

Leask and colleagues push that doubt to its general form. Through SAE stitching they show a larger dictionary does not refine a smaller one so much as learn features it missed, and through meta-autoencoders they show a latent that reads as one concept, “Einstein,” decomposing further into “scientist,” “Germany,” and “prominent figures.” Their flat conclusion: sparse autoencoders do not find a canonical set of features, and no dictionary size is the one to call the model's own.⁹

None of this unwinds the core result. Superposition is a real account of why neurons are polysemantic, and a sparse autoencoder does recover directions more interpretable and more causally precise than the neuron basis or the standard linear alternatives. What the anchor does not establish, and does not claim to, is that these directions are the model's true features or that they beat simple baselines downstream. The honest 2025 position is a promising, contested tool whose evaluation is unfinished.