COMP5318 — Week 7 Supplement
Mathematical Foundations
Perceptron rule, sigmoid derivative, backpropagation deltas for output and hidden neurons, weight-update equations, momentum, Xavier initialisation, softmax and cross-entropy loss.
Perceptron: model and learning rule
A perceptron computes a weighted sum, adds a bias, and applies a step transfer function. Training adjusts \(w\) and \(b\) one example at a time, in the direction of the input scaled by the per-example error.
Core perceptron formulas
Linear combination: \[ n = w \cdot x + b \]
Step transfer: \[ a = f(n) = \begin{cases} 1 & n \ge 0 \\ 0 & n < 0 \end{cases} \]
Per-example error: \[ e = t - a \in \{-1, 0, +1\} \]
Update in matrix form: \[ w^{\text{new}} = w^{\text{old}} + e\,x^\top, \qquad b^{\text{new}} = b^{\text{old}} + e \]
If the training data are linearly separable, the perceptron learning rule terminates after a finite number of updates with a separating hyperplane. If they are not linearly separable (e.g. XOR), the rule never converges.
Worked Example 1: one perceptron update
Take the slide's data: \(x = (1,0,0)\), target \(t = 0\), weights \(w = (0.3, 0.2, 0.4)\), bias \(b = 0.1\).
Compute the output
\[ n = (0.3)(1) + (0.2)(0) + (0.4)(0) + 0.1 = 0.4 \]
\(a = \text{step}(0.4) = 1\), so \(e = t - a = -1\).
Apply the update
\[ w^{\text{new}} = (0.3, 0.2, 0.4) - (1, 0, 0) = (-0.7, 0.2, 0.4) \]
\[ b^{\text{new}} = 0.1 - 1 = -0.9 \]
Repeat the same arithmetic for every training example to finish one epoch.
Backpropagation: sigmoid derivative and delta rules
Backpropagation derives the weight update from the chain rule applied to a smooth activation. With sigmoid, the algebra collapses into two clean delta rules — one for output neurons, one for hidden neurons — which is why the algorithm is hand-computable.
Sigmoid and its derivative
\[ f(z) = \frac{1}{1 + e^{-z}}, \qquad f'(z) = f(z)\,\bigl(1 - f(z)\bigr) \]
Delta values
Output neuron \(q\): \[ \delta_q = (t_q - o_q)\,o_q\,(1 - o_q) \]
Hidden neuron \(q\) with downstream neurons \(i\): \[ \delta_q = o_q\,(1 - o_q)\,\sum_i w_{qi}\,\delta_i \]
Weight and bias updates
Weight from neuron \(p\) to neuron \(q\): \[ w_{pq}^{\text{new}} = w_{pq}^{\text{old}} + \eta\,\delta_q\,o_p \]
Bias of neuron \(q\): \[ \theta_q^{\text{new}} = \theta_q^{\text{old}} + \eta\,\delta_q \]
\(\Delta w_{pq}(t) = \eta\,\delta_q\,o_p + \mu\,(w_{pq}(t) - w_{pq}(t-1))\). The momentum coefficient \(\mu\) propagates the previous step into the current one, reducing oscillation on noisy error surfaces.
Worked Example 2: one backprop iteration
Network with three input neurons, two hidden neurons (\(4,5\)) and one output neuron (\(6\)). Input \(x = (1, 0, 1)\), target \(t = 1\), learning rate \(\eta = 0.9\).
Forward pass
Hidden 4: \(z_4 = 1 \cdot 0.2 + 0 \cdot 0.4 + 1 \cdot (-0.5) - 0.4 = -0.7\), \(o_4 = \sigma(-0.7) \approx 0.332\).
Hidden 5: \(z_5 = -0.3 + 0 + 0.2 + 0.2 = 0.1\), \(o_5 = \sigma(0.1) \approx 0.525\).
Output 6: \(z_6 = 0.332 \cdot (-0.3) + 0.525 \cdot (-0.2) + 0.1 = -0.105\), \(o_6 \approx 0.474\).
Backward pass
Output delta: \(\delta_6 = (1 - 0.474) \cdot 0.474 \cdot (1 - 0.474) \approx 0.1311\).
\(\Delta w_{46} = 0.9 \cdot 0.1311 \cdot 0.332 \approx 0.039\), so \(w_{46}^{\text{new}} \approx -0.261\).
\(\Delta \theta_6 = 0.9 \cdot 0.1311 \approx 0.118\), so \(\theta_6^{\text{new}} \approx 0.218\).
Hidden delta: \(\delta_4 = 0.332 \cdot (1-0.332) \cdot (-0.3) \cdot 0.1311 \approx -0.0087\).
Activations, softmax and cross-entropy
ReLU and leaky ReLU
\[ \text{ReLU}(x) = \max(0, x), \qquad \text{LReLU}(x) = \max(\alpha x, x), \quad \alpha \approx 0.01 \]
Gradient of ReLU is \(1\) for \(x > 0\) and \(0\) for \(x < 0\), so positive paths preserve gradient magnitude across layers.
Softmax
\[ p_i = \frac{e^{o_i}}{\sum_j e^{o_j}} \]
Slide example: \((0.3, 0.8, 0.2) \mapsto (0.28, 0.46, 0.26)\); the resulting probabilities sum to 1.
Categorical cross-entropy
For one-hot label \(y_i\) and softmax prediction \(\hat{y}_i\) of class \(j\): \[ \text{CCE}_i = -\sum_{j=1}^C y_{ij}\,\log \hat{y}_{ij} \]
Summing over \(N\) examples gives the dataset-level loss minimised by training.
Weights of a neuron are drawn from \(\mathcal{N}(0, \sigma^2)\) with \(\sigma = \sqrt{2/(N_\text{in} + N_\text{out})}\). The aim is to keep activation variance approximately constant across layers, which mitigates vanishing/exploding signals at initialisation.