LoRA (low-rank adaptation)

LoRA (low-rank adaptation) is a cheap way to train an existing AI model further on your own examples. It leaves the model itself untouched and trains a small add-on, an adapter, that adjusts how the model behaves. The adapter is a small separate file. That makes it affordable to give one model a separate adapter per task or per customer, for example one for each client's house style.

How does it work?

Ordinary fine-tuning adjusts every weight of a model. LoRA, introduced by Hu et al. in 2021, freezes the original weights and adds to a weight matrix W an update made of two thin matrices, B and A: the layer then uses W + BA. If W measures d × k, then B measures d × r and A measures r × k, with a rank r far smaller than d and k. Only A and B are trained. B starts at zero, so at the start the model behaves exactly like the original.

For large models the saving is big. The weight matrices of GPT-3 175B are 12,288 wide, yet a rank of 1 or 2 was enough on some of the paper's tasks. Compared with full fine-tuning of GPT-3, LoRA trained 10,000 times fewer parameters, cut GPU memory during training from 1.2 TB to 350 GB and trained 25% faster. On RoBERTa, DeBERTa, GPT-2 and GPT-3, it matched or beat full fine-tuning in quality. After training, BA can be added into W, so the model answers as fast as before. LoRA belongs to the PEFT methods (parameter-efficient fine-tuning).

Why use adapters?

• They are small. For GPT-3 175B with rank 4 on the query and value matrices of the attention layers, a checkpoint shrinks from 350 GB to 35 MB. The paper works out that 100 adapted models then take about 354 GB instead of 35 TB.
• One base model can serve many adapters. Merged adapters make it hard to batch requests for different tasks, so serving systems keep them separate: S-LoRA (Sheng et al., 2023) serves thousands of adapters on one or more GPUs.
• Vendors ship features this way. In March 2026, IBM released Granite 4.0 3B Vision as a LoRA adapter on top of its Granite 4.0 Micro language model, so the same deployment serves both image tasks and text-only tasks (IBM).
• They combine with quantization. QLoRA (Dettmers et al., 2023) trains the adapters on top of a 4-bit model. Fine-tuning a 65B model then needs less than 48 GB of GPU memory instead of more than 780 GB, and a 33B model trains on a single consumer GPU.

Does LoRA match full fine-tuning?

Not always. Biderman et al. (2024) trained on programming and mathematics, with about 100,000 instruction examples or 20 billion tokens of further pre-training. In the usual low-rank settings, LoRA learned substantially less than full fine-tuning, but kept more of what the base model could do outside those domains (less catastrophic forgetting).

Thinking Machines (September 2025) found that LoRA performed the same as full fine-tuning on small and medium instruction and reasoning datasets, and in reinforcement learning even at small ranks. It worked best on all weight matrices, especially the MLP and mixture-of-experts layers; LoRA on the attention matrices alone did worse. Its best learning rate was about 10 times that of full fine-tuning. Hugging Face's PEFT library puts LoRA on the query and value matrices by default, and on all linear layers with target_modules="all-linear".

Example

Suppose a web agency writes product texts for twenty webshops, each with its own house style. It trains one LoRA adapter per shop on that shop's approved texts, all on the same open base model. With the paper's figures for a GPT-3-sized model, twenty adapters take 20 × 35 MB = 700 MB, where twenty fully fine-tuned copies would take 20 × 350 GB = 7 TB. The server keeps the base model in memory and uses the right adapter for each request.