Lecture 3: Models — Non-linear

MLP

Non-linearity

• Linear: \(\mathbf{x}W\). Two-layer non-linear: \(\mathbf{x}\mapsto g(\mathbf{x}W_1)W_2\).
• Activations: sigmoid \(\sigma\), tanh, ReLU \(\max(0,x)\).
• Layer = affine + bias + nonlinearity; stack = MLP / feed-forward.
• Non-linear needed for XOR; ReLU net can re-embed points to become linearly separable.

RecallWhy can’t a single linear layer solve XOR? Name three activations.

Optimisation

Gradients & backprop

• Minimise smooth loss (e.g. log-loss) via negative gradient · SGD (noise from minibatch order).
• Logistic: \(\partial \mathcal{L}/\partial W_j = (\sigma(\mathbf{x}\!\cdot\!\mathbf{w})-y)x_j\).
• Deep nets: chain rule on computation graph → automatic differentiation (forward values, backward grads).

RecallWhat does backprop compute? Why chain rule?

Generalisation

Regularisation

• Overfitting risk with flexible nets.
• \(\ell_1\)/\(\ell_2\) penalties on weights.
• Dropout — zero random units in training.
• Early stopping when validation loss plateaus.

RecallTwo regularisers that change the loss; one that changes the forward pass only during training.

Sequences

RNNs

• Mean/concat of word vectors loses order; RNN keeps hidden state \(\mathbf{h}_t\) per step.
• Elman cell: \(\mathbf{h}_t=\tanh(W\mathbf{x}_t+V\mathbf{h}_{t-1}+\mathbf{b}_h)\), \(\mathbf{y}_t=\tanh(U\mathbf{h}_t+\mathbf{b}_y)\).
• Init \(\mathbf{h}_0\): zeros or learned.

RecallElman update: what two terms feed into \(\mathbf{h}_t\)?

Heads

Transducer vs encoder

• Transducer: output per time step (e.g. POS).
• Encoder/acceptor: one label from final \(\mathbf{h}\) (e.g. sentiment).
• Training: BPTT — loss per step or sequence, gradients through unrolled graph.
• Bidirectional RNN: two RNNs (L→R and R→L) whose hidden states are concatenated → full left+right context per position.
Bidirectional RNN: forward (blue) and backward (purple) hidden states concatenated at each position for full context.
• Stacked RNN: output of one RNN layer feeds as input to the next; deeper representations (commonly 2–4 layers).
• POS tagging: assign grammatical category (noun, verb, adj…) per token; classic transducer task. "Time flies like an arrow" → multiple valid POS parses.
• NER: identify named entities (person, location, org) and their types; uses BIO/BIOES tagging scheme (Begin, Inside, Outside, End, Single).
IO
visitedO
TheB-ORG
UniversityI-ORG
ofI-ORG
SydneyE-ORG
inO
AustraliaB-LOC
• Multi-task: single shared RNN with multiple output heads (e.g. POS + NER simultaneously); shared hidden states benefit both tasks.

RecallDifference between transducer and encoder use of RNN output.

Long sequences

Exploding & vanishing gradients

• Exploding: large grads → instability / overflow → gradient clipping.
• Vanishing: repeated \(\tanh\) & multiplications by \(V\) shrink signal over time → hard to learn long-range deps.
• LSTM: forget/input/output gates + additive cell-state path → gradients preserved over long distances; cell state \(\mathbf{c}_t\) does not pass through \(\tanh\).
LSTM cell: forget gate controls what to erase from cₜ; input gate + candidate add new info; output gate controls hₜ. Cell state uses addition — no tanh squashing.
• GRU: simpler variant with update + reset gates; fewer parameters, similar performance.
GRU: reset gate r controls how much of hₜ₋₁ feeds into candidate; update gate z interpolates between old h and new candidate. Fewer gates than LSTM.

RecallOne fix for exploding; why vanishing hurts learning long dependencies.

Lab

Workshop 3

• chapters/chapter3/Materials/Workshop3/workshop3.ipynb — PyTorch tensors, Dataset/DataLoader, mod-*.jsonl text classification.

Quick practice