LoRA ties full fine-tuning where intrinsic rank predicts it should

Full fine-tuning of GPT-3 needed a new 350GB checkpoint per task

Pretraining and adaptation is a two-step trade: spend enormous compute once on a general model, then spend much less adapting it to each task you actually need. The trade breaks down as the general model grows, because the standard way to adapt it, retraining every parameter, scales with the model, not the task. GPT-3 175B fine-tuned on a new task produces a new 175-billion parameter model, a checkpoint on the order of 350GB, and a lab serving ten tasks needs ten of them.¹ Hu and colleagues called this "prohibitively expensive" and set out to adapt the model without retraining most of it.¹

Their answer freezes the pretrained weights entirely and adds a second, much smaller set of trainable matrices alongside them. Whether that works rests on one bet: that the correction a task needs, the difference between the pretrained weights and the fine-tuned ones, does not use anywhere near the full range of directions the weight matrix could in principle move in. The paper calls this the intrinsic rank of the update, and everything else in the method, and everything the follow-on record has since tested, hangs on whether that bet holds for the task in front of you.¹

A frozen weight gains one low-rank correction

Take one weight matrix inside a transformer layer, shape d \times k. Ordinary fine-tuning learns a full update \Delta W of that same shape, free to move in any of its d \times k directions. LoRA replaces that update with the product of two much smaller matrices, B (shape d \times r) and A (shape r \times k), and picks r far smaller than d or k. The forward pass adds their product to the frozen weight's own output.

Rank ablations in the paper push r down as far as it will go before quality drops. On the datasets they tested, a rank of one is already enough to adapt the query and value projections. GPT-3 configurations in the paper range from r=1, the smallest budget tested, up through r=8 and beyond, and GPT-2 configurations run r=4 to r=16.¹ At that rank the saving compounds into the paper's own headline number. A full fine-tuned GPT-3 checkpoint runs about 350GB, or 350,000MB. The trained low-rank matrices for the same model take up about 35MB. 350,000 divided by 35 is 10,000, the ten-thousand-fold reduction the abstract claims.¹

A rank that low working at all rests on what the pretrained weights already contain. A model trained on trillions of tokens has already built most of the machinery a downstream task needs. Adapting it to a narrower task is closer to nudging existing representations into a slightly different arrangement than building new capability from nothing, and a nudge, unlike new capability, does not need the full space of possible weight changes to express it. The intrinsic rank hypothesis says that nudge is low-dimensional, and the rank ablations are the direct test of exactly how low.¹

What actually beat full fine-tuning in 2021

Every one of these is a classification, generation, or text-to-structure task where the answer stays close to the pretrained model's existing competence: parse a question into SQL, judge entailment, summarize a chat, generate text conditioned on a short table. None asks the model to build a skill it did not already have in some form. At r=1, LoRA already reaches that competence on two of the three GPT-3 benchmarks above, MultiNLI and SAMSum, and comes within 0.4 points of it on the third, WikiSQL. A larger, 37.7M-parameter configuration closes even that gap, reaching 74.0% against full fine-tuning's 73.8%.¹

The abstract's other claim, no added inference latency, follows directly from the same equation: once training ends, add BA into W_0 and the model is one matrix again, the same shape it always was.¹ That merge is a deliberate step, not something that happens on its own. The reference implementation ships a merge function for exactly this reason,² and Hugging Face's PEFT library, now the main way practitioners run LoRA, documents calling merge_and_unload() before serving.³ Skip that step, which a deployment serving many task-specific adapters off one shared base model has good reason to do, and inference costs two matrix multiplications instead of one. The latency claim and the multi-adapter convenience that made LoRA attractive for serving are not in tension in principle. They are in tension for exactly the deployment that wants both at once.

Quantizing the frozen half changes nothing about the trained half

If the correction a task needs really is low-rank, it should not much matter how precisely the frozen weights it is being added to are stored, because the adapter is not encoding new information about those weights, only the small nudge on top of them. QLoRA is the test of that prediction: quantize the pretrained model to 4-bit precision, keep it frozen, and train LoRA matrices in higher precision on top.⁴ Across more than a thousand trained models, the paper reports that this preserves full 16-bit fine-tuning task performance despite the base weights carrying a quarter of the information they started with.⁴ Guanaco, a 65-billion-parameter model trained this way in 24 hours on a single 48GB GPU, reached 99.3% of ChatGPT's score on the paper's benchmark.⁴

QLoRA is an extension of LoRA, not an independent check on it; Dettmers and colleagues built directly on the original method rather than setting out to test its assumptions.⁴ But the result still bears on the intrinsic-rank claim, because quantization is exactly the kind of change that would hurt a method if the adaptation depended on precise access to the frozen weights. It didn't. The low-rank correction held up under a four-fold compression of the thing it was correcting, which is more evidence that the correction lives in a small subspace largely decoupled from how the base weights are represented.

Code and math need more rank than LoRA offers

The original paper never tested a task where the model has to learn something closer to a new skill than a nudge on an old one. Biderman and colleagues did, in a 2024 study accepted to Transactions on Machine Learning Research with Featured Certification, continuing pretraining and instruction-tuning models on code and math and comparing LoRA against full fine-tuning at matched compute.⁵ The parity the 2021 benchmarks showed did not travel.

Take the worst case: 0.175 against a full fine-tuning score of 0.263 on code continued pretraining. Divide the gap by the full fine-tuning score, 0.088 ÷ 0.263, and LoRA lands about a third short.⁵ Instruction tuning narrows the gap, and math narrows it further still, down to a three percent shortfall on GSM8K. But the direction never reverses. The closer a task sits to the model's existing competence, the smaller the shortfall, and continued pretraining on an unfamiliar domain sits furthest from that competence of anything either paper tested.

Biderman and colleagues measured why directly. They estimated the effective rank of the weight changes full fine-tuning actually makes on these tasks and found it running 2,000 to 4,096, by their 90-percent-variance threshold.⁵ Even LoRA's largest rank in the study, r=256, sits well under that. 4,096 divided by 256 is 16, and 2,000 divided by 256 is about 8. So the best-resourced LoRA configuration tested was still eight to sixteen times short of the rank full fine-tuning was actually using. The more typical, smaller ranks practitioners reach for fall short by the fuller ten-to-hundred-fold margin the study reports.⁵ This is the same intrinsic-rank quantity the original paper measured at one to sixteen on GLUE-style tasks. Neither number is wrong; the quantity itself scales with what the task demands, and nothing in either paper predicts it in advance. A practitioner still has to find it by ablation, same as Hu and colleagues did in 2021.¹

Where the adapter sits matters as much as how large it is. Biderman and colleagues tested LoRA applied only to attention matrices, only to the MLP, and to all of a model's linear layers at matched parameter budgets, and attention-only trailed the other two by a wide margin across every domain.⁵ The reason is not that attention matters less. It's that restricting the update to one class of matrix caps how many independent directions the whole model can move in, whatever rank each individual matrix carries, and code and math evidently need more of those directions than the attention matrices alone can supply. Spreading the same trainable budget across more matrices approximates a higher effective rank than concentrating it in fewer places does.

One comparison caveat belongs here. The paper itself specifies random Gaussian initialization for A and zero for B, so training starts exactly at the pretrained weights,¹ but the reference implementation initializes A with Kaiming-uniform in code, a default the paper's own text never mentions.² A 2025 study found that default has convergence properties data-aware alternatives improve on.⁶ Initialization is an uncontrolled variable in every LoRA-versus- full-fine-tuning comparison run so far, this one included.

A structured-pruning study looks at first glance like it shows the adapters themselves are over-parameterized. It doesn't. LoRAShear prunes the frozen base model, guided by which parameters a LoRA module marks as barely used for the task, and cuts the base LLM's own footprint by 20% for a 1% quality loss.⁷ That is evidence LoRA is a useful lens on where a large pretrained model's capacity goes unused, not evidence about how much capacity the adapter itself needs.

Less capacity bought less forgetting

The same study that measured where LoRA loses also measured what it keeps. After continued pretraining on code, models tuned with LoRA held onto more of their performance on general-knowledge benchmarks, HellaSwag, ARC-Challenge, and WinoGrande, than fully fine-tuned models did. On the instruction-tuning runs the retained score was 0.63 against full fine-tuning's 0.45. Divide one by the other, 0.63 ÷ 0.45, and LoRA kept 40% more of the source domain's performance.⁵

Read the two findings together and there is no contradiction to resolve, only a trade the original paper's benchmarks never surfaced because none of them tested it. A rank ceiling caps how far training can push a model away from where it started, and that cap works in both directions at once. It limits the move away from the pretrained weights toward the new task, and it limits the move away from whatever the pretrained model already knew. Full fine-tuning has no such ceiling, so it can chase a hard target further and forget more of what came before while doing it.⁵ LoRA is not a cheaper way to reach the same destination on every task. On the tasks where it falls behind, it is a regularizer that reaches a worse destination and forgets less getting there, and a practitioner choosing between them is choosing which of those two outcomes the deployment can afford.

The gaps LoRA does carry are not fixed features of low-rank adaptation itself, only of the specific decomposition Hu and colleagues chose. DoRA splits a weight matrix into magnitude and direction and applies LoRA only to the direction term. Across commonsense-reasoning benchmarks it beats vanilla LoRA on every model size tested: 78.4% against 74.7% on LLaMA-7B, 85.2% against 80.8% on LLaMA3-8B.⁸ The gains are real and consistent, and modest, one to four and a half points, not the difference full fine-tuning opens on code and math. That is the right scale for what DoRA is. It is evidence that the 2021 recipe was one point in a wider design space, not the ceiling of what a low-rank update can do, and not a reversal of the harder limit continued pretraining exposes.

The condition the 2021 claim depends on