LoRA & parameter-efficient fine-tuning

A 4x4 target update, approximated by the product of two length-4 vectors — 8 numbers standing in for 16. Step through training and watch that low-rank approximation converge onto the real target.

LoRA — short for Low-Rank Adaptation — freezes a pretrained weight matrix $W$ and represents the fine-tuning update as a low-rank product instead of a full matrix:

• $W$ — the original pretrained weight matrix, kept frozen throughout fine-tuning.
• $W'$ — the effective weight matrix used after applying the fine-tuning update.
• $\Delta W$ — the fine-tuning update itself, decomposed into a low-rank product rather than stored as a full matrix.
• $B$ — one trainable low-rank factor, of shape $n \times r$.
• $A$ — the other trainable low-rank factor, of shape $r \times n$.
• $n$ — the size of the original (square) weight matrix.
• $r$ — the rank of the update, chosen much smaller than $n$.

1. Parameter cost drops from quadratic to linear

   A full update to an $n\times n$ matrix costs $n^2$ trainable parameters. A rank-$r$ update costs only $2nr$. For a $4096\times4096$ attention matrix and $r=8$:

   • Full update: $n^2 = 4096^2 = 16{,}777{,}216$ parameters
   • LoRA update: $2nr = 2 \times 4096 \times 8 = 65{,}536$ parameters
   • Ratio: $65{,}536 / 16{,}777{,}216 \approx 0.0039 = 0.39\%$ — under half a percent
2. The catch: rank is capped at r

   $\Delta W$ can only ever be rank $\leq r$, however training goes. If the update a task actually needs has higher intrinsic rank than $r$, LoRA simply cannot represent it exactly.

The target update, the full fine-tune's result after one step, and LoRA's result after 100 steps — visually indistinguishable. Full fine-tuning used 16 parameters and one step; LoRA used 8 parameters and a hundred steps to land in the same place.

A 4x4 target update that happens to be exactly rank-1 — $\Delta W^* = a^*b^{*\top}$ for some vectors $a^*, b^*$:

1. Full fine-tuning: trivial, but 16 parameters

   Each of the 16 entries is its own independent quadratic loss, same shape as every "ideal learning rate" example from earlier in this course. At $\eta=0.5$, a full update matches the target exactly in one step.

2. LoRA: 8 parameters, but a harder optimization

   $\Delta W \approx ab^\top$ makes the loss a product of two vectors, not a sum of independent terms — no one-step closed form. After 100 gradient steps at $\eta=0.02$, the loss drops to $\approx6\times10^{-31}$: effectively exact, at half the parameter count.

3. The learned vectors aren't unique — and that's fine

   The trained $a, b$ don't match $a^*, b^*$ directly — scaling $a$ up and $b$ down by the same factor leaves $ab^\top$ unchanged, so infinitely many $(a,b)$ pairs solve this exactly. What matters is the product, and that converges to the true update regardless of which particular pair training lands on.