Math Foundations — Lecture 3

Affine transforms, activations, chain rule, SGD, regularisation, Elman RNN, gradient problems.

Affine transformation

\[\mathbf{z} = \mathbf{x}\,W + \mathbf{b}\]

\(\mathbf{x}\in\mathbb{R}^{d_\text{in}}\), \(W\in\mathbb{R}^{d_\text{in}\times d_\text{out}}\), \(\mathbf{b}\in\mathbb{R}^{d_\text{out}}\). A layer = affine + activation.

Activation functions

Sigmoid

\[\sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma'(z)=\sigma(z)(1-\sigma(z))\]

Output in \((0,1)\). Derivative max at \(z=0\) (\(0.25\)); saturates → vanishing grad.

Tanh

\[\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\quad \tanh'(z)=1-\tanh^2(z)\]

Output in \((-1,1)\). Zero-centred (helps optimisation vs sigmoid). Still saturates at extremes.

ReLU

\[\mathrm{ReLU}(z)=\max(0,z),\quad \mathrm{ReLU}'(z)=\begin{cases}1&z>0\\0&z\le 0\end{cases}\]

No saturation for positive inputs; sparse activation. "Dead neuron" if always \(z\le 0\).

Quick Check: Which activation avoids saturation for positive inputs?

Sigmoid

Tanh

ReLU

MLP (multi-layer perceptron)

Two-layer example:

\[\mathbf{h} = g(\mathbf{x}\,W_1+\mathbf{b}_1),\qquad \hat{\mathbf{y}} = \mathbf{h}\,W_2+\mathbf{b}_2\]

Without \(g\), composition collapses: \(\mathbf{x}W_1 W_2=\mathbf{x}W'\) — still linear. Non-linearity \(g\) is essential (XOR example).

Gradient descent & SGD

\[\theta \;\leftarrow\; \theta - \eta\,\nabla_\theta\mathcal{L}(\theta)\]

\(\eta\) = learning rate. SGD: compute gradient on a random minibatch instead of full dataset; introduces noise that can help escape local minima.

Chain rule & backpropagation

For a composition \(f = f_3 \circ f_2 \circ f_1\), the chain rule gives:

\[\frac{\partial \mathcal{L}}{\partial \theta_1} = \frac{\partial \mathcal{L}}{\partial f_3}\;\frac{\partial f_3}{\partial f_2}\;\frac{\partial f_2}{\partial f_1}\;\frac{\partial f_1}{\partial \theta_1}\]

Forward pass: compute and cache intermediate values. Backward pass: propagate gradients from loss to each parameter via cached Jacobians.

Logistic example

\[\frac{\partial\mathcal{L}}{\partial w_j}=(\sigma(\mathbf{x}\cdot\mathbf{w})-y)\,x_j\]

Quick Check: In backprop, which pass uses cached intermediate values?

Forward pass

Backward pass

Regularisation

\(\ell_2\) (weight decay)

\[\mathcal{L}_\text{reg}=\mathcal{L}+\lambda\sum_i w_i^2\]

Penalises large weights; equivalent to Gaussian prior. Gradient contribution: \(+2\lambda w_i\).

\(\ell_1\) (sparsity)

\[\mathcal{L}_\text{reg}=\mathcal{L}+\lambda\sum_i |w_i|\]

Encourages exact zeros → feature selection.

Dropout

At training time, zero each hidden unit independently with probability \(p\). At test time, multiply weights by \((1-p)\) (or scale during training). Does not change the loss function — modifies forward pass only.

Elman RNN

\[\mathbf{h}_t = \tanh\!\bigl(W\,\mathbf{x}_t + V\,\mathbf{h}_{t-1}+\mathbf{b}_h\bigr)\] \[\mathbf{y}_t = \tanh\!\bigl(U\,\mathbf{h}_t + \mathbf{b}_y\bigr)\]

\(\mathbf{x}_t\)Input at step \(t\) (e.g. word embedding).

\(\mathbf{h}_t\)Hidden state — carries information from previous steps.

\(W,V,U\)Shared weight matrices (input-to-hidden, hidden-to-hidden, hidden-to-output).

Initialise \(\mathbf{h}_0\) to zeros or learn it as a parameter.

BPTT (backpropagation through time)

Unroll the RNN for \(T\) steps → apply chain rule across all steps. Total gradient for a parameter sums contributions from each time step.

Vanishing & exploding gradients

Gradient of loss w.r.t. \(\mathbf{h}_k\) involves the product:

\[\prod_{t=k+1}^{T}\frac{\partial \mathbf{h}_t}{\partial \mathbf{h}_{t-1}} = \prod_{t=k+1}^{T}\mathrm{diag}\!\bigl(\tanh'(\cdot)\bigr)\,V\]

If \(\|V\|\) and \(\tanh'\) are <1 repeatedly → product shrinks exponentially (vanishing).

If \(\|V\|\) is large → product grows exponentially (exploding).

Fixes

Gradient clipping: cap \(\|\nabla\|\) to a max norm. Gated architectures (LSTM, GRU): additive paths that preserve gradients over long distances — covered in later lectures.

Quick Check: Which problem does gradient clipping address?

Vanishing

Exploding