Math Foundations — Lecture 9
Pretraining objective, full FT cost, low-rank decomposition for LoRA, parameter counts, QLoRA quantisation.
Pretraining objective
A causal language model with parameters \(\theta\) is trained by minimising the negative log-likelihood of the next token across a corpus \(\mathcal{D}\):
For masked LMs (BERT) the inner sum runs over masked positions only. Either way the loss is computed on raw text — no labels needed, so pretraining can use trillions of tokens.
Full finetuning — memory cost
Let \(N\) be the total parameter count. Per-parameter memory during Adam training (mixed precision, naïve):
That's ~16 bytes/parameter just for weights + optimiser. For \(N=7\times 10^9\): \(\sim 112\) GB — well beyond a single 24 GB GPU.
LoRA — low-rank decomposition
Take any frozen weight matrix \(W_0 \in \mathbb{R}^{d \times d}\) (assume square for simplicity). LoRA writes the finetuning update as a product of two narrow factors:
The forward pass becomes
Only \(A\) and \(B\) are trainable; \(W_0\) stays frozen. By construction \(\operatorname{rank}(\Delta W) \le r\), encoding the assumption that the useful finetuning update lies in a low-dimensional subspace.
Initialisation
Choose \(A \sim \mathcal{N}(0, \sigma^2)\) and \(B = 0\). Then at step 0:
i.e. the model output is identical to the pretrained model — training departs smoothly from the foundation.
Optional scaling
The factor \(\alpha/r\) (LoRA scaling) decouples learning-rate effects from the choice of rank \(r\).
Parameter count: full FT vs LoRA
One \(d \times d\) projection inside a Transformer layer.
Ratio of LoRA to full FT for one matrix:
Concrete numbers for typical Transformer projection sizes:
| \(d\) | \(r\) | Full FT (\(d^2\)) | LoRA (\(2dr\)) | Ratio |
|---|---|---|---|---|
| 768 | 8 | 589{,}824 | 12{,}288 | ~48× smaller |
| 4096 | 8 | 16{,}777{,}216 | 65{,}536 | ~256× smaller |
| 4096 | 16 | 16{,}777{,}216 | 131{,}072 | ~128× smaller |
| 8192 | 16 | 67{,}108{,}864 | 262{,}144 | ~256× smaller |
Across an entire 7B model, LoRA on attention projections typically yields <0.1% of \(N\) as trainable parameters — small enough that optimiser state and adapter checkpoints become trivial.
Inference: merging the delta
After training, define
Replace \(W_0\) with \(W_{\text{merged}}\) and discard \(A, B\). The model is now indistinguishable from a fully fine-tuned model at the level of one matrix — zero added FLOPs and zero added latency at inference. (Compare adapters/prefix tuning, which insert extra computation that cannot be folded away.)
QLoRA — quantising the base
QLoRA stores the frozen base weights in 4-bit precision. A real-valued weight \(w\) is approximated by
where \(\{c_i\}\) are 16 fixed codebook values and \(s\) is a per-block scale. The NF4 codebook places the \(c_i\) at quantiles of a unit normal, matching the empirical weight distribution of pretrained Transformers.
Memory ratio versus fp16 storage of the same model:
During training: dequantise on the fly inside each kernel, run the LoRA bypass \(B A\) in bf16, accumulate gradients only into \(A, B\). The base never updates, so its 4-bit quantisation never degrades. Combined with paged Adam moments, a 65B model is trainable on a single 48 GB GPU.
Summary: parameter footprint
Per Transformer projection matrix of size \(d \times d\):