COMP5318 — Applied Machine Learning
Week 9: Attention and Transformers
Sequence-to-sequence RNN encoder-decoders compress an entire sentence into one context vector — a bottleneck. The attention mechanism removes that bottleneck by letting the decoder look back at every encoder state. The Transformer (Vaswani et al., 2017) takes attention to its logical conclusion: drop recurrence entirely and build the whole network from self-attention and feed-forward layers, with positional encoding to inject word order.
Course materials
Week 9 layers attention on top of the seq2seq RNN from last week and then replaces recurrence entirely with the Transformer. The slides follow Jay Alammar's "Illustrated Transformer" — read the Q/K/V figure carefully, then the multi-head and positional-encoding figures, and finally the full encoder-decoder stack.
| File | What you should learn from it |
|---|---|
| ml9.pdf | RNN applications (VQA, reading comprehension, seq2seq for translation and chat); the seq2seq encoder-decoder with attention; the Transformer (Vaswani et al., 2017): self-attention with Query/Key/Value, scaled dot-product attention, multi-head attention, positional encoding, residual connections, and the full encoder-decoder stack; why Transformers beat RNNs (parallelism, long-range dependencies). |
From seq2seq to attention
Recurrent networks shine on sequences but struggle to translate long sentences. The classical fix is an encoder-decoder: one RNN reads the source, another generates the target. Attention then removes the bottleneck of squeezing all source information into one vector.
Two RNNs in series. The encoder reads the source sentence and emits a final hidden state — a context vector meant to summarise the whole input. The decoder starts from that context and generates the target one token at a time, feeding its own previous output back as the next input.
RNN applications
The same encoder-decoder template covers many sequence tasks:
- Language translation ("我是一个学生" → "I am a student")
- Chat / dialogue (reply generation)
- Visual Question Answering (image + question → word answer)
- Reading comprehension (facts + query → answer)
The bottleneck problem
The encoder must compress an entire variable-length sentence into one fixed-size vector. For long sentences this loses information — the decoder gets only a blurred summary. Translation quality drops with input length.
Instead of relying on one context vector, the decoder at each step computes a weighted combination of every encoder hidden state. A learned scoring function decides how much to attend to each source position. The weights change at every decoder step, so different output words can focus on different source words.
The Transformer architecture
Vaswani et al. (2017), "Attention Is All You Need", asked: if attention is the key ingredient, do we need RNNs at all? The Transformer replaces recurrence entirely with stacked self-attention and feed-forward layers — both inside the encoder and the decoder.
Encoder stack
A stack of \(N\) identical encoder layers. Each layer has two sub-layers: multi-head self-attention and a position-wise feed-forward network. Each sub-layer has a residual connection and layer norm.
Decoder stack
A stack of \(N\) identical decoder layers. Each has three sub-layers: masked self-attention (so position \(t\) cannot peek at future tokens), encoder-decoder attention, and a feed-forward block.
Embeddings + positions
Words are turned into vectors by an embedding lookup, then a positional encoding is added so the otherwise position-agnostic self-attention layer can use word order.
Linear + softmax
The top of the decoder is projected to a vocabulary-sized vector by a final linear layer and softmaxed to produce next-token probabilities. Training uses cross-entropy / KL.
Self-attention: Query, Key, Value
Self-attention lets each word in a sentence look at every other word and decide which ones are relevant for encoding it. In the sentence "The animal didn't cross the street because it was too tired", self-attention learns that "it" should attend to "animal".
For each input embedding \(x_i\) we produce three vectors by learned linear projections: \(q_i = x_i W^Q\), \(k_i = x_i W^K\), \(v_i = x_i W^V\). The matrices \(W^Q, W^K, W^V\) are the only new parameters.
Six steps of self-attention
- Project each input to a Query, Key, Value vector.
- Score \(s_{ij} = q_i \cdot k_j\) — how relevant word \(j\) is to word \(i\).
- Scale by \(\sqrt{d_k}\) for stable gradients.
- Softmax along \(j\) to get weights that sum to 1.
- Multiply each \(v_j\) by its softmax weight.
- Sum the weighted values → output for position \(i\).
Matrix form
Pack all embeddings into a matrix \(X\). Then \(Q = XW^Q\), \(K = XW^K\), \(V = XW^V\), and the entire self-attention output is computed in one go:
\(\text{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\dfrac{QK^\top}{\sqrt{d_k}}\right) V\)
Why scale by \(\sqrt{d_k}\)?
For large key dimension \(d_k\), the dot products \(q \cdot k\) have large variance, pushing softmax into saturated regions where gradients vanish. Dividing by \(\sqrt{d_k}\) keeps the variance bounded.
What softmax does
Turns the row of scores into a probability distribution over source positions, so the output is a convex combination of the value vectors — focused on the relevant words, drowning out the rest.
Self vs. cross-attention
In self-attention, Q, K, V come from the same sequence. In encoder-decoder attention, Q comes from the decoder and K, V come from the encoder output — so the decoder queries the encoder.
Multi-head attention and positional encoding
One self-attention layer can only pay attention in one way. Multi-head attention runs several attention "heads" in parallel, each with its own \(W^Q, W^K, W^V\), so the model can attend to different things at once.
Run \(h\) parallel attention heads with independent projections, producing outputs \(\text{head}_1, \ldots, \text{head}_h\). Concatenate them and project through a final matrix \(W^O\):
\(\text{MultiHead}(Q,K,V) = \mathrm{Concat}(\text{head}_1, \ldots, \text{head}_h)\,W^O\)
Why multiple heads?
Each head learns a different "representation subspace". One head might track syntactic dependencies (subject — verb), another co-reference ("it" — "animal"), a third long-range topical similarity. Concatenation gives the next layer access to all of them.
Dimensions add up
With model dimension \(d_{\text{model}}\) and \(h\) heads, each head usually uses \(d_k = d_v = d_{\text{model}}/h\) so the total parameter count matches a single-head layer of width \(d_{\text{model}}\).
Self-attention is permutation-equivariant: shuffle the input tokens and the output shuffles the same way. To inject word order, a positional encoding vector is added to each input embedding before the first encoder layer. Vaswani et al. use fixed sinusoids of different frequencies so the model can attend to relative positions.
Residuals & layer norm
Each sub-layer in the encoder/decoder is wrapped in a residual connection followed by layer normalisation: \(\text{LayerNorm}(x + \text{Sublayer}(x))\). Helps optimisation in deep stacks.
Masked decoder attention
During training, the decoder self-attention is masked so position \(t\) sees only positions \(\le t\). This preserves the autoregressive property while still letting the whole sequence be processed in parallel.
Loss function
The decoder emits a probability distribution over the vocabulary at each step. Training minimises cross-entropy (equivalently KL divergence) against the one-hot target distribution.
Study questions to answer before moving on
Explain in one or two sentences why attention helps translation quality on long sentences, where a plain seq2seq encoder-decoder struggles.
Plain seq2seq compresses the whole input into one fixed-length context vector — for long sentences this loses information. Attention lets the decoder at every output step query all encoder hidden states, so detail from any source position can flow directly into the relevant target word without going through the bottleneck.
Given embeddings packed into \(X\), write down the scaled dot-product self-attention output and explain what each matrix does.
Project: \(Q = XW^Q\), \(K = XW^K\), \(V = XW^V\). Then \[ \text{Attention}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) V. \] \(QK^\top\) is the matrix of query-key similarity scores; dividing by \(\sqrt{d_k}\) keeps the variance bounded so softmax doesn't saturate; softmax turns each row into a distribution over source positions; multiplying by \(V\) returns a weighted combination of value vectors.
(a) Why use multiple attention heads instead of one with a larger \(d_k\)? (b) Why are positional encodings necessary in a Transformer but not in an RNN?
(a) Each head can attend in a different way — syntactic dependencies, co-reference, topical similarity, etc. Concatenating gives the next layer access to all "representation subspaces" simultaneously, whereas a single bigger head can only produce one weighted sum per position.
(b) An RNN processes tokens sequentially, so order is implicit in the order of computation. A Transformer processes all tokens in parallel and pure self-attention is permutation-equivariant, so it has no built-in notion of position. Adding a positional encoding to each token embedding restores that signal.
Chapter quizzes
Self-test and math questions for this week are in the Quiz Hub (practice or exam mode).