Lecture 6: Models — Transformer

Beyond RNN

Why move beyond RNN encoders?

• Enc–dec RNNs: information bottleneck in one vector; tokens processed strictly sequentially.
• Attention: each output step builds context as weighted sum of encoder states (or embeddings) → less bottleneck, more parallelism.

RecallWhat two RNN limits does attention mitigate?

Scores

Forms of attention

• Dot-product: \(e = s^\top h\).
• Bilinear: \(e = s^\top W h\).
• Reduced-rank factorisations of bilinear.
• Additive (Bahdanau): \(e = b^\top \tanh(W_1 h + W_2 s)\).
• Scaled dot-product: \(e = q^\top k / \sqrt{d_k}\) — dot magnitudes grow with \(d_k\); scaling keeps softmax from saturating (gradients healthier).
• View: key–value lookup — queries attend to keys; blend values.

RecallWhy divide by \(\sqrt{d_k}\)?

All-to-all

Self-attention

• Inputs \(\mathbf{x}_i\): scores \(e_{ij} = \mathbf{x}_i^\top \mathbf{x}_j\), softmax over \(j\), then \(\mathbf{o}_i = \sum_j \alpha_{ij} \mathbf{x}_j\).
• Each position gets a contextual vector from all positions.

RecallSelf-attention output \(\mathbf{o}_i\) as weighted sum — weights from what?

Learned

Query, key, value projections

• Learn \(Q, K, V\): \(\mathbf{q}_i = Q\tilde{\mathbf{x}}_i\), \(\mathbf{k}_j = K\tilde{\mathbf{x}}_j\), \(\mathbf{v}_j = V\tilde{\mathbf{x}}_j\).
• \(\alpha_{ij} = \mathrm{softmax}_j(\mathbf{q}_i^\top \mathbf{k}_j / \sqrt{d_k})\), \(\mathbf{t}_i = \sum_j \alpha_{ij}\mathbf{v}_j\); often + feed-forward.
• Multi-head: parallel heads (different subspaces) → concat — heads can specialise (syntax vs entities).

RecallWhat do multiple heads buy you vs one head?

Order

Position and causal masking

• Without position info, self-attention is permutation-invariant.
• Add positional encodings \(\mathbf{p}_i\): learned per index, sinusoidal, or RoPE (rotary) — relative cosine similarity for same-distance pairs.
• Decoder LM: causal mask — \(e_{ij}=-\infty\) for \(j>i\) so position \(i\) cannot attend to future tokens.

RecallWhy causal mask in decoder training?

Stack

Transformer encoder layer

• Self-attention sublayer + position-wise FFN (e.g. ReLU MLP).
• Residual connections (add input to sublayer output — smoother gradients).
• Layer norm: \(\mathrm{LN}(\mathbf{x})=\gamma\odot\frac{\mathbf{x}-\mu}{\sigma+\epsilon}+\beta\); \(\mu,\sigma\) over feature dim per token; learned \(\gamma,\beta\).
• Original: \(N\) repeated layers (e.g. 6), multiple heads per layer.

RecallResidual + LN: one purpose each.

Seq2seq

Decoder and cross-attention

• Decoder: masked self-attention (autoregressive) + cross-attention.
• Cross-attention: queries from decoder; keys/values from encoder output — seq2seq path without single fixed-length bottleneck vector.

RecallWhere do Q vs K,V come from in cross-attention?

Impact

Landmark models & scaling

• BERT (2018): encoder-only Transformer + WordPiece + masked LM + next-sentence prediction → state-of-the-art on GLUE benchmarks.
• GPT-1 (2018): decoder-only Transformer + BPE + causal LM → strong zero/few-shot via prompting.
• Scaling laws (Kaplan 2020; Chinchilla 2022): performance improves as a power law with compute, data, and parameters — guides cost/quality trade-offs.
• Training costs range from ~$500 (small) to ~$10M+ (frontier); most university labs cannot train from scratch.
• Efficiency: self-attention is \(O(n^2)\) in sequence length; efficient variants (Longformer, LinFormer, FlashAttention) reduce memory or exploit sparsity.
• Modern models (e.g. Llama): BPE tokenisation, RoPE, carefully mixed data, multiple sizes.

RecallBERT vs GPT: encoder-only or decoder-only? What do scaling laws predict?

Lab

Workshop 7 & materials

• chapters/chapter6/Materials/Workshop7/workshop7.ipynb — char or word seq2seq; optional attention on Fr–En pairs (bridge from Lec 5 RNN to attention here).

Quick practice