LoRA, short for low-rank adaptation, is a parameter-efficient fine-tuning method that adapts a large pretrained model by freezing its original weights and training only small pairs of low-rank matrices added to selected layers. In AI model training, it is the most widely used way to specialize language models and diffusion models without the memory and storage cost of full fine-tuning.
A typical LoRA adapter trains well under 1 percent of the model’s parameters and is stored as a separate file of a few megabytes to a few hundred megabytes. The same base model can therefore serve many tasks by loading different adapters.
How LoRA modifies a weight matrix
In full fine-tuning, every weight matrix W of the model is updated. LoRA instead keeps W frozen and learns an update expressed as the product of two thin matrices, B and A. If W has size d by k, A has size r by k and B has size d by r, where the rank r is small, often between 4 and 64. The effective weight becomes W plus B times A, scaled by a factor.
The idea rests on the observation that the change needed to adapt a model tends to have low intrinsic rank: a few directions in weight space capture most of what a new task requires. For a 4,096 by 4,096 matrix, full fine-tuning trains about 16.8 million values, while LoRA with rank 8 trains about 65,000, a reduction of roughly 256 times for that layer.
B is initialized to zero, so training starts from the unchanged base model. After training, the update can be merged into W, which removes any extra inference latency, or kept separate so that adapters can be swapped.
Key LoRA hyperparameters
LoRA has a small set of hyperparameters, and their defaults work reasonably for many tasks. The table summarizes common ranges and their effect.
| Hyperparameter | Typical range | Effect |
|---|---|---|
| Rank r | 4 to 64, sometimes 128 or more | Capacity of the adaptation |
| Alpha | Often equal to r or twice r | Scales the update, scale is alpha / r |
| Target modules | Attention projections, MLP layers | Which weights are adapted |
| Learning rate | About 1e-4 to 3e-4 | Higher than full fine-tuning |
| Dropout | 0 to 0.1 | Regularization on the adapter path |
| Epochs | 1 to 5 for most datasets | Risk of overfitting small data |
Applying LoRA to all linear layers, including the feed-forward blocks, usually works better than adapting only the attention query and value projections used in early experiments. Raising the rank helps with large or very different target tasks but gives diminishing returns for narrow ones.
LoRA compared with other training methods
LoRA sits among several options for adapting a model. The table compares it with its closest neighbors in the training toolbox.
| Method | What is trained | Memory need | Typical use |
|---|---|---|---|
| Full fine-tuning | All weights | Very high | Maximum quality, ample hardware |
| LoRA | Low-rank adapters | Moderate | Most domain and style adaptation |
| QLoRA | Adapters on 4-bit frozen base | Low | Large models on one GPU |
| Prompt tuning | Soft prompt vectors | Very low | Light task steering |
| Distillation | A smaller student model | Depends on student | Cheaper deployment |
LoRA is a mechanism for how weights are updated, not a training objective. It can be combined with supervised fine-tuning, RLHF or DPO. QLoRA pairs it with quantization of the frozen base model, which makes adapting models with tens of billions of parameters possible on a single high-memory GPU.
Several refinements build on the same idea. DoRA separates the magnitude and direction of weight updates, and other variants adjust ranks per layer. None has fully displaced the original method, which remains the default in most libraries.
When to choose LoRA
LoRA is the default choice when hardware, storage or the number of variants is a constraint. The decision table covers common situations.
| Situation | Recommended choice | Reason |
|---|---|---|
| One base model, many customers | Separate LoRA adapters | Small files, hot-swappable |
| Limited GPU memory | QLoRA | Quantized base fits in memory |
| New language or large knowledge shift | Full fine-tuning or continued pretraining | Low rank may limit capacity |
| Style or format consistency | LoRA with low rank | Small change, little data |
| Image style or subject in diffusion | LoRA on the denoiser | Fast training, easy sharing |
Limitations and common mistakes
LoRA makes adaptation cheap, but cheap training does not guarantee good results. Data quality and evaluation still decide the outcome.
- Knowledge injection is weak: LoRA is better at teaching behavior, format and style than large volumes of new facts.
- Overfitting small datasets: a few hundred examples trained for many epochs can make the model repeat training phrases.
- Base model mismatch: an adapter only works with the exact base model version it was trained on.
- Merged adapters interact: combining several adapters can produce unpredictable behavior.
- Catastrophic forgetting is reduced, not removed: general abilities can still degrade.
A held-out evaluation set that measures both the target task and general capabilities is the most reliable safeguard.
Stripped to the essentials
A large AI model is like a thick textbook that took enormous effort to write. Rewriting the whole book for every new job would be slow and expensive. LoRA instead adds a thin set of sticky notes that adjust how the book is read for a particular purpose.
The original book stays untouched, and the notes are small enough to swap in seconds. One model can then handle many specialized jobs simply by switching which set of notes is attached.
Where it is applied
A software company wants its internal assistant to write support replies in its own tone and to follow a strict response template. Full fine-tuning of its 8-billion-parameter open model would require several large GPUs. The team instead trains a LoRA adapter with rank 16 on all linear layers, using 6,000 reviewed past replies.
Training on a single GPU takes a few hours, and the adapter file is roughly 80 megabytes. The team evaluates it on 400 held-out tickets and a general reasoning test to make sure it has not lost broader skills. Later, the legal and sales departments train their own adapters on the same base model.
The serving platform loads the shared base once and attaches the right adapter per request, so three specialized assistants run for the hardware cost of one.