Lecture 5: Models — Encoder–Decoder

Vectors

Static embeddings: word2vec, GloVe, FastText

• word2vec CBOW: predict centre word from averaged context embeddings.
• Skip-gram: predict context from centre word.
• GloVe: global co-occurrence stats + learned dot-product objective.
• FastText: word = sum of character n-gram vectors → rare / misspelled words.
• Domain ≠ pretraining corpus (e.g. medical vs web): fine-tune on in-domain data or init from published vectors and continue training.

RecallCBOW vs skip-gram direction; what FastText adds vs word2vec.

Sense

Contextual representations

• One static vector per word → cannot split senses (bat animal vs sport).
• Bidirectional RNNs: left + right context per position → contextual vectors per token.
• Training: CBOW-style prediction at each position (masked words).
• ELMo (2018): landmark deep contextualised representations.

RecallWhy static embeddings fail on polysemy; what ELMo signals.

Seq2seq

Encoder–decoder

• Encoder RNN reads source → hidden states; decoder generates target L→R conditioned on encoder.
• Uses: MT, summarisation, dialogue, code generation.
• Stopping: fixed length, or until </s> (EOS).
• Practice: decoder often gets encoder context every step; encoder can be bidirectional; stacks of layers common.

RecallTwo ways to end generation.

Train vs test

Training and inference

• Teacher forcing: feed decoder the gold previous token in training so errors don’t compound immediately.
• Test: greedy argmax, sampling, or beam search (Lecture 4).
• Compare hypotheses of different lengths: normalise scores (e.g. by length).

Left: decoder receives gold tokens. Right: decoder feeds back its own predictions.

RecallWhat is fed to the decoder during training vs at inference?

Metrics

Evaluating translation

• Gold: human judgements (fluency, adequacy).
• chrF: character n-gram precision/recall, combined with \(F_\beta\) (tune \(\beta\) precision vs recall).
• BLEU: word n-gram precisions, geometric mean, brevity penalty.
• Tokenisation affects scores (e.g. read vs Reading).
• chrF example: character 3-gram matching → TP=11, FP=3, FN=5 → P=11/14≈0.79, R=11/16≈0.69, F1≈0.73.

RecallchrF vs BLEU: what unit each uses; why BLEU has brevity penalty.

Subwords

Tokenisation and BPE

• Whitespace-only: misses contractions, punctuation attachment, rare words.
• BPE: start from characters; repeatedly merge most frequent adjacent pair until vocab size \(K\) (e.g. 100k).
• Subwords generalise to unseen typos and long compounds.
• WordPiece: similar to BPE but merges by perplexity reduction (not raw frequency); used by BERT.
• BPE at inference: apply learned merge rules in order to segment new text (e.g. "store" → "s t o r e" → merge pairs greedily).

RecallBPE merge rule in one sentence.

Bridge

Encoder–decoder limits

• Classic seq2seq: source compressed into one vector (bottleneck).
• Decoder sequential → hard to parallelise.
• Attention fix: at each decoder step compute scores \(e_{ij}=\mathbf{s}_i^\top\mathbf{h}_j\), softmax → weights \(\alpha_{ij}\), blend encoder states → context \(\mathbf{a}_i=\sum_j\alpha_{ij}\mathbf{h}_j\).
• Variants introduced here: dot-product, bilinear \(e=\mathbf{s}^\top W\mathbf{h}\), additive \(e=\mathbf{b}^\top\tanh(W_1\mathbf{h}+W_2\mathbf{s})\), scaled \(e=\mathbf{s}^\top\mathbf{h}/\sqrt{d}\).

RecallTwo limits of vanilla encoder–decoder that attention addresses.

Lab

Workshop 5 & materials

• chapters/chapter5/Materials/Workshop5/workshop5.ipynb — Keras TextVectorization, IMDB sentiment, Conv1D, LSTM/BiLSTM pipelines.

Quick practice