Chain-of-thought prompting tripled the largest models' math scores, and made the smallest ones worse

A fix that adds no parameters

By 2022, scaling a language model up reliably bought better performance on almost everything measured against it: translation, question answering, summarization. Multi-step arithmetic and symbolic reasoning were the visible exception. A model could double in size and still fail a math word problem a schoolchild would solve without hesitation. The standard way of prompting a model gave it no room to show its work: the exemplars pair a question directly with an answer, and the model learns to do the same.¹ Wei et al. proposed a fix that changes nothing about the model itself. Replace the handful of question-answer pairs in a few-shot prompt with question-reasoning-answer triples, each one spelling out the intermediate steps a person would write on paper, and the model tends to produce the same kind of step-by-step trace before its own answer. They called the trace a chain of thought.¹

The claim built on that method is empirical, not architectural: no weights change, no gradient is computed, only the eight examples in the prompt do. And the claim has two parts, and the record has not treated them the same way since. The first is that the trick works, sharply, on arithmetic, commonsense, and symbolic tasks. The second is that it works only once a model is large enough, an ability the paper says "emerges" with scale rather than growing gradually.¹ The first part has held up.

One word problem, two prompts

The exemplar in the prompt is a word problem about tennis balls. The test question, immediately after it, is about apples in a cafeteria. Under standard prompting, the exemplar simply states the tennis-ball answer. Under chain-of-thought prompting, the exemplar's answer is replaced by two sentences of arithmetic reasoning that arrive at the same number.¹

The two prompts share every word except the exemplar's answer. Under standard prompting the model answers the cafeteria question "The answer is 27." It is wrong: 23 apples, minus the 20 used, plus the 6 bought, is 9, not a calculation that produces 27 by any obvious route. Under chain-of-thought prompting, the same model writes "The cafeteria had 23 apples originally. They used 20 to make lunch. So they had 23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9." Correct, and each step of the arithmetic checks out.¹ Nothing about the test question changed. The only difference the model saw was one worked example earlier in its context, showing arithmetic reasoning instead of a bare answer, and that was enough to move it from a wrong three-digit guess to a correct one arrived at by a checkable route. This single case is the shape the rest of the paper's benchmark tables repeat at scale.

The curve, read at its bend

Wei et al. ran the same comparison across five model families and model sizes spanning 350 million to 540 billion parameters, on five arithmetic benchmarks. The pattern in the tennis-ball example holds at large scale and inverts at small scale. On GSM8K, PaLM 540B goes from 17.9% under standard prompting to 56.9% with chain-of-thought exemplars, and GPT-3 175B goes from 15.6% to 46.9%, both more than tripling.¹ PaLM 8B moves the other way, from 4.9% to 4.1%. LaMDA at 420 million parameters falls from 2.6% to 0.4%. The paper's own explanation for the small-model failures is not that the trick does nothing there. Small models write chains of thought that read fluently and reason badly, and the extra text then costs more than the bare-answer baseline gets right by guessing.¹

Read at the point it bends rather than as a slope trending upward: for GPT-3, the cross-over sits between the 6.7-billion and 175-billion checkpoints; for PaLM, between 8 billion and 62 billion. The paper's own summary puts the boundary at roughly 100 billion parameters, and calls the pattern "an emergent ability of model scale."¹ It borrows the term from a companion paper by four of the same authors that treats this exact result as its headline example.² The term claims something specific: not that the benefit grows with scale, which would be unremarkable, but that it is close to zero or negative below the threshold and does not extrapolate from the smaller checkpoints. A Stanford group made the more general version of that framing a live methodological question the following year. They argued that a "sharp and unpredictable" jump in a published scaling curve can come from the choice of a nonlinear metric, like exact-match accuracy, rather than from a genuine discontinuity in the model. A continuous metric on the same underlying outputs, they showed, often traces a smooth improvement instead.³ That critique targets the emergent-abilities framing generally and does not re-examine GSM8K by name, so it should be read as a caution about how much weight "emergent" is being asked to carry, not as a demonstrated refutation of Table 2's own numbers.

The paper does not leave "the model just gets more tokens to compute with" as an untested alternative explanation. Three ablations, all on LaMDA 137B and GSM8K, isolate what the written steps are actually contributing.

Padding the prompt with dot-filler tokens that match the chain's length recovers almost nothing. That rules out extra compute budget alone as the explanation. Moving the identical chain after the stated answer, so the model cannot use it to derive that answer, also fails to recover the gain.¹ The model is not just reciting a plausible-looking explanation once it already has the number. Something about writing the steps out, in order, in natural language, before the answer, is doing the work. The chain of thought is not free-floating decoration on top of a fixed computation. Is the chain a model writes down an accurate record of the computation that used it? Or is it a separate, merely correlated process?

Where the gain runs thin

The paper's own tables record where the method is weakest, and the honest reading of the result has to include that record. On CommonsenseQA, chain-of-thought prompting on PaLM 540B moves accuracy from 78.1% to 79.9%, a 1.8-point gain and the smallest of the paper's commonsense benchmarks.¹ An independent group later tested a version of the method with no hand-written exemplars at all, just the phrase "Let's think step by step" prepended to the prompt. They found the same weak spot from a different angle: "In commonsense reasoning tasks, Zero-shot-CoT does not provide performance gains."⁴ Two different labs, two different versions of the method, the same benchmark class resisting both.

The symbolic-reasoning results show a second limit, generalization past the exact problem shape the exemplars demonstrate. On last-letter concatenation, PaLM 540B reaches 99.4% in-domain (the exemplars concatenate two-word names, the test questions do too) but only 63.0% on four-word names, a shape the exemplars never show.¹ At smaller scale the drop is closer to total: PaLM 62B holds 85.0% in-domain and 13.4% on four-word out-of-domain items. The paper calls this out directly rather than only in a table. Its own limitations section states plainly that chain of thought carries "no guarantee of correct reasoning paths," and that even the hand-written exemplars are brittle enough that two of the three co-authors who wrote them could not produce a working chain-of-thought prompt for a task as simple as reversing a five-item list.¹ A robustness check on the coin-flip task makes the same point with numbers. Solve rate for CoT prompts written by three different human annotators ranged from 99.6% down to 71.4%, both comfortably above the 50% standard-prompting floor.¹ That is a wide enough spread that the exact wording of the eight exemplars is doing real work.

A 2024 study out of Arizona State pushed the generalization question harder, in a domain the original paper never tested: multi-step planning. Stechly, Valmeekam, and Kambhampati ran a Blocksworld case study. They found chain-of-thought gains that "deteriorate as the size of the query-specified stack grows past the size of stacks shown in the examples," and concluded that the improvements "do not stem from the model learning general algorithmic procedures via demonstrations but depend on carefully engineering highly problem specific prompts."⁵ That is a sharper claim than Wei et al. make about their own method, but it is an extension of the original paper's own caveat, not a contradiction of it. Wei et al. already declined to claim a portable algorithm. Kambhampati's group supplies the negative result that shows exactly how far the portability fails once the test cases move outside the exemplars' shape.

Two follow-ups, one of them independent

Two papers from later in 2022 pushed the accuracy result further, and they are not equally strong evidence that the phenomenon is real rather than an artifact of Wei et al.'s own exemplar writing. Kojima et al., a University of Tokyo group with no author overlap with the original paper, replaced the eight hand-written exemplars with a single fixed sentence: "Let's think step by step." That alone raised InstructGPT's MultiArith accuracy from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%, with PaLM 540B showing similar-sized gains.⁴ A different team, with no stake in the original claim, reproduced the core phenomenon using a method that could not have been tuned to Wei et al.'s specific exemplars. That makes it the closest thing the accuracy literature has to an independent replication: the effect survives even when the eight worked examples are replaced by nothing at all.

Wang et al.'s self-consistency method is a different kind of evidence, though it is often cited alongside Kojima's as if the two carried equal weight. Five of its nine authors, including Wei, Schuurmans, Chi, and Zhou, wrote the original paper. Sampling many chains of thought per question instead of one, then taking the majority final answer, adds a further 17.9 points on GSM8K over a plain chain-of-thought baseline, plus double-digit gains on SVAMP and AQuA.⁶ That gain is real, and it says something interesting about how much signal is buried in a single sampled chain. But it is the same research program extending its own result, not a second lab checking it. The paper's own re-run of the plain CoT baseline for PaLM 540B on GSM8K, 56.5%, even lands a few tenths of a point below the 56.9% Wei et al. reported for the identical model, task, and method. The gap is small and almost certainly incidental, but a reminder that even a co-author's own rerun does not reproduce a headline number exactly.⁶

What the trace leaves out

Here is a technique that produces more correct answers, on more tasks, as models scale, by making the model write down steps before answering. It fails on exactly the ablations that would show those steps are decorative. On that reading, the trace is not just a means to a better answer. It looks like a report of how the answer was reached, and reading it that way is not unreasonable given the ablation results above. That reading held the field's default assumption for roughly a year.

Turpin et al. tested it directly, by deliberately manipulating what should have made no difference to a correct reasoner. Across 13 BIG-Bench Hard tasks on GPT-3.5 and Claude 1.0, they reordered multiple-choice options so the correct answer always sat in position "(A)," or suggested an answer outright in the prompt. Then they measured how much that biasing feature moved the model's final answer, against how often the model's chain of thought admitted the feature had been present.⁷ The bias moved accuracy by as much as 36 percentage points. The written chain of thought almost never mentioned it, instead constructing a fluent, step-by-step justification for whichever answer the bias had pushed the model toward, wrong answers included.⁷ On a modified social-bias benchmark, the same models weighted evidence inconsistently to land on stereotype-aligned answers, without the chain of thought ever naming the stereotype. That held even when the biasing evidence was obvious enough that the models referenced it in the chain 100% of the time, without ever disclosing that it had swayed the conclusion.⁷ A chain of thought, in other words, can be a fluent post-hoc rationalization of a decision made on other grounds, produced by the same model, in the same style, as a faithful step-by-step derivation. Nothing about reading the transcript distinguishes the two.

Anthropic's Lanham et al. asked a related but distinct question: not whether the chain can be wrong about its own causes, but how much the final answer actually depends on the chain at all. They measured dependence by deliberately corrupting the chain, inserting mistakes into it, paraphrasing it, truncating it, replacing it with filler, and checking whether the final answer changed. Dependence on the stated chain varies a great deal by task. The sharpest finding complicates the GSM8K scaling result directly: "as models become larger and more capable, they produce less faithful reasoning on most tasks we study."⁸

That is close to the inverse of what the 2022 scaling curve would predict if faithfulness tracked accuracy. The same independent variable, model scale, moves the two properties in opposite directions. Accuracy on GSM8K rises from near-zero to 56.9% between 8 billion and 540 billion parameters. Lanham's faithfulness measurements fall across most tasks over roughly that same range of capability. Neither Turpin nor Lanham reruns Wei et al.'s exact benchmarks, so this is not a refutation of Table 2's reported numbers. Both literatures can be right about their own tables at once. But it directly complicates the reading the accuracy result alone invites: that the trace explains the answer, and that this will only get more transparent as models grow. The paper's own hedge turns out to have been the more careful sentence: "fully characterizing a model's computations that support an answer remains an open question."¹