Efficient adaptation
LoRA: reshaping a 100B model without retraining it — the math of low-rank updates and when full fine-tuning still wins.
You have a base model. You want it to write in your product's voice, follow your domain's conventions, refuse what your users need it to refuse. Full fine-tuning means training every parameter — storage and compute for 100B weights, per customer, per variant. LoRA (Low-Rank Adaptation) is the trick that made adaptation a product decision instead of a lab project.
Start here
Fine-tuning changes each weight matrix W by a correction ΔW. LoRA's bet: the useful corrections are low-dimensional — so represent ΔW as the product of two skinny matrices A·B with rank r ≪ d, freeze W, and train only A and B. Under 1% of the parameters, most of the effect.
The math
For a weight matrix (using ):
Trainable parameters: full FT has ; LoRA has — at that is parameters, ≈0.4% of the matrix (and under 1% of a full model's parameters is typical). Scale it:
One weight matrix W (4096×4096 = 16.7M params) — full FT vs LoRA
bar scaled 4× so the LoRA bar is visible at all
the usual sweet spot: ~0.1–0.4% of params here, most of the adaptability — LoRA freezes W and trains only A (d×r) and B (r×d); their product approximates the full-rank update ΔW.
Real arithmetic for one layer; a real adaptation targets a subset of layers. Bonus: adapters can be swapped at serving time — one base model, many personalities.
Why this works, honestly: the hypothesis — not a guarantee — is that task-specific updates live in a low-rank subspace ("a style change doesn't need to rewrite all 16.7M numbers independently"). Empirically it holds well enough across domains that LoRA became the default. At low ranks capacity does bottleneck: too-small r means the adapter cannot express the behavior change.
The serving trick that made it commercial
is a sum of two matrix products. At serving time you can keep one base model in memory and many adapters, swapping terms per request (or merging selected adapters into a copy of the weights). One checkpoint serves many customized variants — this is the infrastructure behind most "fine-tuned model" products. Swap some:
One base model, many personalities
14 GB + 4×0.06 GB vs 4×14 GB full fine-tunes
Because W′ = W + BA and only BA changes, serving swaps the small BA term per request — one 14 GB base plus60-MB adapters instead of four full checkpoints (~56 GB). Real deployment (S-LoRA and successors) batches hundreds of adapters against one base GPU-resident. Numbers: stated assumptions, not a specific product.
Note
Same distinction from the RAG lesson, sharpened: LoRA adapts behavior; RAG supplies facts. They compose — a LoRA-tuned customer support model retrieving from your docs is a standard production pattern.
When full fine-tuning still wins
LoRA is the default; it is not universal:
- New knowledge ingestion — teaching large amounts of new domain knowledge generally needs full-capacity updates or continued pretraining, not rank-8 corrections.
- Training from scratch on new objectives — alignment runs (the post-training chapter) operate at full rank.
- Distillation — training a smaller student to mimic a larger teacher is its own full training run; efficient adaptation adapts, it doesn't shrink.
The adaptation menu, complete
| Goal | Tool |
|---|---|
| Model lacks facts | RAG (retrieve at inference) |
| Model behaves wrong (style, format, refusals) | LoRA / full SFT |
| Model's judgments need alignment | preference tuning (RLHF/DPO) |
| Task is checkable | RLVR |
| Model too slow/costly to serve | distillation, quantization |
Real product decisions usually combine two or three of these. Knowing which axis a problem lives on — knowledge, behavior, judgment, or cost — is the practitioner's core skill.
Next: limits, safety, and what's next — where all this still breaks, what safety engineering looks like, and where the frontier is heading. After that, the interpretability chapter opens the box.