COMP5318 — Applied Machine Learning
Week 7: Feedforward Neural Networks
The first deep-learning week shifts from classical models to neural networks. It starts from the perceptron and its linear-separability limit, then builds up the multi-layer perceptron and the backpropagation algorithm, then explains the modern engineering moves that make deep networks trainable in practice: ReLU activations, dropout, softmax outputs, and cross-entropy loss.
Course materials
The Week 7 lecture is dense: it covers the perceptron and the full backpropagation derivation in one go, and finishes with the modern techniques that turn small networks into deep ones. Read the slides alongside this guide, and re-trace the numerical backprop example by hand.
| File | What you should learn from it |
|---|---|
| ml7.pdf | Perceptron model and rule, AND/OR/NAND/XOR analysis, multi-layer perceptron architecture, forward and backward pass, worked backprop example, learning rate / momentum / weight init, vanishing gradients, ReLU, dropout, softmax, cross-entropy. |
The perceptron: a single neuron with a step decision
A perceptron is one neuron with a step transfer function: it sums weighted inputs plus a bias and emits 0 or 1. It is the simplest neural network and the conceptual base for everything that follows.
Given input \(x\), weights \(w\), and bias \(b\), the perceptron computes \(n = w \cdot x + b\) and outputs \(a = \text{step}(n)\): 1 if \(n \ge 0\), else 0. The slide's worked example: \(x=(0.2,0.3)\), \(w=(2,1)\), \(b=-1.5\) gives \(a = \text{step}(-0.8) = 0\).
Perceptron learning rule
Define the error \(e = t - a\) between the target and the actual output. Then update in matrix form:
\(w^{\text{new}} = w^{\text{old}} + e\,x^\top\) and \(b^{\text{new}} = b^{\text{old}} + e\).
If \(e=0\) the weights do not move; if \(e=+1\) the weights grow towards \(x\); if \(e=-1\) they shrink away from it.
One epoch
An epoch is one full pass through the training set where weights are updated after every example. The end-of-epoch check (all examples correctly classified, or max epochs reached) does not itself change weights.
The lecture is precise about this: the check pass does not count as another epoch.
AND / OR / NAND
Linearly separable in 2D: a single line can separate the positive corner(s) from the negatives. A perceptron solves all three.
XOR
The positives \((0,1)\) and \((1,0)\) sit on opposite sides of the diagonal; no single line separates the classes. A single perceptron cannot represent XOR.
Two layers fix XOR
XOR factors as a combination of NAND, OR, and AND gates. A 2-layer perceptron expresses that combination and solves XOR.
Multi-layer perceptron and backpropagation
A multi-layer perceptron stacks layers of neurons with smooth activation functions so that gradients can flow. The backpropagation algorithm trains the network: one forward pass to produce a prediction, one backward pass to push the error back through the layers, layer by layer.
Layers: input, one or more hidden, output. Feedforward means each neuron only takes input from the previous layer. Fully connected means every neuron in the current layer is connected to every neuron in the previous one.
Sigmoid transfer
For backpropagation to work the activation must be differentiable. The classical choice is the sigmoid \(f(z) = 1/(1+e^{-z})\), whose derivative \(f'(z) = f(z)\,(1-f(z))\) is convenient for hand calculation.
Two passes
Forward: compute every neuron's output. Backward: compute output \(\delta\) from the target, propagate \(\delta\) into the hidden layers, then update each weight as \(\Delta w_{pq} = \eta\,\delta_q\,o_p\).
Output-neuron error
For a sigmoid output neuron: \(\delta_q = (t_q - o_q)\,o_q\,(1-o_q)\). The first factor pulls the prediction toward the target; the second factor vanishes when the unit saturates.
Hidden-neuron error
\(\delta_q = o_q\,(1-o_q)\,\sum_i w_{qi}\,\delta_i\), summed over neurons \(i\) in the layer above. The earlier layers inherit their gradient from the layer that follows them.
Weight update
\(w_{pq}^{\text{new}} = w_{pq}^{\text{old}} + \eta\,\delta_q\,o_p\). The bias updates as \(\theta_q^{\text{new}} = \theta_q^{\text{old}} + \eta\,\delta_q\).
Why "back"-propagation? The output layer is the only place where you actually know the target. Backprop reuses each layer's \(\delta\) to compute the previous layer's \(\delta\) via the weights between them, so the gradient is calculated once per weight rather than re-derived from scratch.
SGD
Stochastic gradient descent updates weights after every single training example. This is the variant the worked lecture example uses.
Mini-batch
Sum the gradients over a small batch, then update once. Smooths SGD noise and uses vectorised hardware efficiently.
Full-batch
Sum over all examples before each update. Stable but slow; rare in practice for large datasets.
Universal approximator
One hidden layer suffices to approximate any continuous function (Cybenko 1989, Hornik et al. 1989). This is an existence theorem, not a construction.
Design choices and hyperparameters
Once the architecture is fixed, performance lives or dies by a few hyperparameters. The lecture walks through each one and gives practical tuning advice.
Layer sizing
Input neurons: one per numerical feature; for a k-valued categorical attribute use one-hot encoding with k binary inputs. Output neurons: one per class for k-class problems (one-hot), or a single sigmoid neuron for binary.
Hidden layers
Tuned by trial and error: too few neurons under-fit, too many over-fit. The lecture suggests growing the hidden layer — start small, train until error plateaus, then add neurons.
Learning rate \(\eta\)
Too small → slow convergence. Too large → oscillation and overshoot. There is no single optimal value; the lecture warns you cannot pick it before training because the error surface changes as you go.
Decay schedules
Time-based: \(\eta_n = \eta_{n-1}/(1+d\,n)\). Exponential: \(\eta_n = \eta_0 e^{-d n}\). Start high to escape shallow regions, decay to settle near a minimum.
Momentum
\(\Delta w_{pq}(t) = \eta\,\delta_q\,o_p + \mu\,(w_{pq}(t) - w_{pq}(t-1))\). The momentum term \(\mu\) carries the previous step forward, damping oscillations and allowing slightly larger \(\eta\).
Random small values in e.g. \([-1,1]\) is the classical default. Xavier initialisation samples from \(\mathcal{N}(0,\sigma^2)\) with \(\sigma = \sqrt{2/(N_\text{in}+N_\text{out})}\), where \(N_\text{in}\) and \(N_\text{out}\) are the fan-in and fan-out of the current neuron. The aim is to keep the variance of activations and gradients stable across layers.
Modern techniques that make deep networks trainable
Deep networks suffered from the vanishing-gradient problem for decades. The combination of better activations, regularisation, output transformations and loss functions made depth practical.
Vanishing gradient: with sigmoid, when a hidden output approaches 0 or 1 the factor \(o\,(1-o)\) is near zero. As gradients are multiplied through many layers, the product collapses to near zero. The early layers receive almost no signal and learn extremely slowly.
ReLU
\(y = \max(0, x)\). Gradient is 1 for \(x > 0\), so positive activations do not saturate. The default for hidden layers in modern deep networks.
Leaky ReLU
\(y = \max(\alpha x, x)\) with small \(\alpha\) (e.g. 0.01). Keeps a non-zero gradient on the negative side so units do not "die".
Dropout
During training, randomly disable a fraction \(p\) of neurons each iteration. The remaining sub-network is forced to be robust; spurious features that depend on specific units are penalised.
Test-time dropout
At inference no neurons are dropped, but the weights are scaled to compensate for the higher expected sum. The result behaves like an ensemble of thinned sub-networks.
Softmax outputs
To interpret the output layer as a probability distribution use \(p_i = e^{o_i}/\sum_j e^{o_j}\). Example from the lecture: outputs \((0.3,0.8,0.2)\) become probabilities approximately \((0.28,0.46,0.26)\), which sum to 1.
Cross-entropy loss
For one-hot labels \(y_i\) and softmax predictions \(\hat{y}_i\) the categorical cross entropy is \(\text{CCE}_i = -\sum_j y_{ij}\,\log \hat{y}_{ij}\). It replaces MSE for classification and penalises confidently wrong predictions much more.
Study questions to answer before moving on
Train a perceptron by hand on the data \((x_1,x_2,x_3) \to t\): \((1,0,0) \to 0\), \((1,0,1) \to 1\), \((1,1,0) \to 0\). Initial weights \(w=(0.3,0.2,0.4)\), bias \(b=0.1\). What weights does the rule produce after one epoch?
Ex.1 \(a=\text{step}(0.4)=1\), \(e=-1\), so \(w \leftarrow w - x = (-0.7,0.2,0.4)\), \(b \leftarrow b - 1 = -0.9\).
Ex.2 \(a=\text{step}(-1.2)=0\), \(e=1\), so \(w \leftarrow w + x = (0.3,0.2,1.4)\), \(b \leftarrow -0.9 + 1 = 0.1\).
Ex.3 \(a=\text{step}(0.6)=1\), \(e=-1\), so \(w \leftarrow w - x = (-0.7,-0.8,1.4)\), \(b \leftarrow 0.1 - 1 = -0.9\).
End of epoch 1: \(w = (-0.7,-0.8,1.4)\), \(b=-0.9\). A check pass shows Example 2 is still wrong, so training continues.
In the lecture's worked example a forward pass gives hidden outputs \(o_4 = 0.332\), \(o_5 = 0.525\) and output \(o_6 = 0.474\) with target \(t=1\) and \(\eta=0.9\). What is \(\delta_6\) and the new weight \(w_{46}\) from old value \(-0.3\)?
\(\delta_6 = (1 - 0.474) \cdot 0.474 \cdot (1 - 0.474) \approx 0.1311\). \(\Delta w_{46} = 0.9 \cdot 0.1311 \cdot 0.332 \approx 0.039\), so \(w_{46}^{\text{new}} \approx -0.261\). The output bias updates to \(\theta_6 \approx 0.218\).
Explain why replacing sigmoid hidden activations with ReLU helps train deeper networks. Reference both the gradient formula and the multiplicative chain through layers.
Sigmoid's derivative \(o(1-o)\) is at most \(0.25\), and is close to 0 whenever a unit saturates. Multiplying many such factors across deep layers shrinks the gradient toward zero (the vanishing-gradient problem), so early layers barely learn. ReLU's gradient is exactly 1 for \(x > 0\), so positive paths preserve gradient magnitude through depth.
Chapter quizzes
Self-test and math questions for this week are in the Quiz Hub (practice or exam mode).