Lecture 9: Training — Pretraining, Finetuning & PEFT

Foundation

Pretraining vs finetuning

• Pretraining: train a large model on huge unlabelled text with next-token prediction (or masked LM). Goal: a general foundation model whose hidden states encode broad linguistic and world knowledge.
• Cost is enormous — frontier runs spend millions of GPU-hours; most labs reuse a public checkpoint (Llama, Mistral, BERT, GPT-2) instead of training from scratch.
• Finetuning: continue training the pretrained model on a smaller task- or domain-specific dataset to adapt behaviour (sentiment, summarisation, code, an instruction style).
• Key intuition: pretraining learns the language, finetuning teaches the job.

RecallWhat objective is used in pretraining a decoder-only LM, and what changes during finetuning?

All weights move

Full finetuning

• Standard recipe pre-2021: backprop through every parameter in the model, store a fresh optimiser state per parameter, save a fresh checkpoint per task.
• For a 7B model in mixed precision: weights ~14 GB, Adam moments ~28 GB, activations & gradients on top — does not fit on a single consumer GPU.
• Risks: catastrophic forgetting of pretraining knowledge if the new data is narrow; per-task checkpoints multiply storage.
• Still the gold standard when data is plentiful, compute is cheap, and you need maximum quality on one task.

RecallWhy does full FT on a 7B model usually need multi-GPU memory?

Tune few, freeze many

Parameter-Efficient Fine-Tuning (PEFT)

• PEFT idea: freeze the pretrained weights and learn a small set of new parameters that nudge model behaviour. Often <1% of the original parameter count.
• Benefits: tiny optimiser state, small checkpoints (swap a few MB per task), much less GPU memory, faster training, easier multi-task deployment.
• Empirical finding: for most downstream tasks, the update a finetuned model applies to its weights has low intrinsic rank — you don't need to move every direction in weight space.
• Families: LoRA (low-rank deltas), adapters (small inserted MLPs), prompt / prefix tuning (learn tokens, not weights).

RecallWhat property of the finetuning update justifies PEFT?

Low-rank deltas

LoRA — Low-Rank Adaptation

• Frozen base weight \(W_0 \in \mathbb{R}^{d \times d}\). Add a learnable delta factorised into two narrow matrices: \(\Delta W = B A\) with \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times d}\), and rank \(r \ll d\).
• Forward pass: \(h = W_0 x + \Delta W\,x = W_0 x + B A\,x\). Only \(A\) and \(B\) receive gradients; \(W_0\) is frozen.
• Parameter count drops from \(d^2\) per matrix to \(2 d r\). For \(d=4096, r=8\): \(\sim 2^{24}\) vs \(\sim 6.5\times 10^4\) — a ~250× reduction.
• Init: \(A\) random Gaussian, \(B = 0\) so \(\Delta W = 0\) at start — finetuning begins from the exact pretrained function.
• Inference: optionally merge \(W_0 + BA\) back into a single dense matrix — zero added latency after merge.
• Usually applied to attention projections (\(W_Q, W_V\), sometimes \(W_K, W_O\) and MLPs).

RecallWrite the trainable parameter count of one LoRA-adapted \(d \times d\) matrix.

Quantise the base

QLoRA — LoRA on a quantised model

• Idea: keep the frozen base in low-precision (e.g. 4-bit NF4 quantisation) and train LoRA adapters in higher precision (bf16) on top.
• Memory wins compound: a 65B model that needs >780 GB in fp16 fits in ~48 GB at 4-bit — finetune frontier-scale models on a single GPU.
• Tricks: NF4 data type (normal-distribution-shaped quantiles), double quantisation (quantise the quantisation constants), paged optimisers (offload Adam moments to CPU when out of memory).
• Quality: matches 16-bit full FT on benchmarks while using a fraction of the memory.
• Related: LLM.int8 (Dettmers 2022) already showed that you can run inference in 8-bit with outlier-aware mixed precision — QLoRA pushes the same idea to training.

RecallWhy is QLoRA more memory-efficient than vanilla LoRA?

Other PEFT families

Adapters, prompt tuning, prefix tuning

• Adapters (Houlsby 2019): insert small bottleneck MLPs (down-project to \(r\), non-linearity, up-project) inside each Transformer block. Only adapter weights train. Adds inference latency (extra layers); newer variants (parallel/AdapterFusion) mitigate this.
• Prompt tuning (Lester 2021): prepend a few learnable embedding vectors to the input, freeze everything else. Tiny parameter count; works well only at large model scales.
• Prefix tuning (Li & Liang 2021): learn key/value prefix vectors at every Transformer layer (not just the input). Stronger than prompt tuning, smaller than LoRA.
• Common thread: where you inject extra capacity (input tokens vs every attention K/V vs every weight matrix) trades off expressiveness against parameter count.

RecallPrompt tuning vs prefix tuning: what is added, and where?

Practical recipe

When to use what

• Lots of data, lots of GPUs, one target task: full finetuning — usually the highest ceiling.
• Limited GPU memory, base model fits in fp16: LoRA with \(r=4\)–\(16\) on attention projections. Cheap, mergeable at inference, near-FT quality on most tasks.
• Limited memory and base is too big for fp16: QLoRA — quantise base to 4-bit, train LoRA on top, optionally page Adam moments to CPU.
• Many tasks, must hot-swap: adapters or LoRA — ship one base + a folder of small adapter files; load per request.
• Latency-critical inference: prefer LoRA (merge into \(W_0\)) over adapters/prefix (extra runtime).
• Very little labelled data: in-context learning / few-shot prompting before any training — only escalate to PEFT when prompting plateaus.

RecallWhy is LoRA preferred over adapters when inference latency matters?

Lab

Workshop & materials

• chapters/chapter9/materials/lecture-9.pdf — original handout slides (efficiency & in-context learning).
• HuggingFace peft library: \(\sim\)10 lines wrap any model with LoRA / QLoRA / prompt-tuning configs.
• Try: load a 7B base in 4-bit (bitsandbytes), attach a LoRA config (r=8, target q_proj,v_proj), finetune on a small instruction set, then merge and export.

Quick practice