COMP5318 — Applied Machine Learning
Week 8: Convolutional and Recurrent Networks
Two specialised neural-network families. Convolutional networks exploit the 2D spatial structure of images with learned filters, strides, and pooling. Recurrent networks unroll over time to handle sequences, training via backpropagation through time, and LSTM cells use gated memory to preserve long-range information.
Course materials
Week 8 bundles two large topics. Read the CNN half with image examples in mind; read the RNN half thinking about sequences such as text and time series. The LSTM cell diagram is the densest figure in the slides — re-draw it from memory before moving on.
| File | What you should learn from it |
|---|---|
| ml8.pdf | CNN motivation, convolution with filters, stride and padding, pooling, full CNN architecture (alternating conv + pool + FC layers), multi-channel filters; RNN motivation for sequences, simple Elman RNN, unrolling, backpropagation through time, character/word prediction; LSTM cell with forget/input/output gates and cell-state update. |
Convolutional networks: filters, stride, and pooling
A CNN is a feedforward neural network trained with backpropagation, but designed to exploit the spatial structure of images. Pixels nearby are usually related; a CNN encodes that prior with small, learnable filters that slide across the image.
A small filter (also called kernel, e.g. 3×3) slides across the input image. At each position the filter is multiplied element-wise with the underlying pixels and the products are summed (plus a bias). The result is a single value in the feature map. The region under the filter is the receptive field.
Filters as feature detectors
A hand-designed filter could detect a curve, an edge, or a corner. In a CNN the filter values are learned by backpropagation — the network discovers the features it needs from data.
One feature map per filter; many filters produce many feature maps, each highlighting a different pattern.
Stride and padding
Stride \((s_h, s_v)\): how many pixels the filter shifts each step. Bigger strides → smaller feature maps. Padding: add zero pixels around the border so the filter can reach the edge without going off the image.
Pooling
A sub-sampling layer that summarises a region with one value. Max pooling takes the maximum; average pooling takes the mean. Reduces spatial size and adds robustness to small translations.
Multi-channel input
Colour images have three channels (R, G, B). The filter becomes 3D: its third dimension matches the channel count, so the dot product is over both space and channels.
Why not a plain MLP?
A fully connected layer flattens the image into a vector and loses spatial structure. CNNs preserve spatial locality, share weights via filter reuse, and have far fewer parameters than an equivalently expressive MLP.
Putting a CNN together
A typical CNN alternates convolutional layers (often with ReLU) and pooling layers to extract a hierarchy of features, then ends with one or two fully connected layers and a softmax over classes.
Conv layer
Many learnable filters produce a stack of feature maps. Often followed by a non-linear activation such as ReLU.
Pool layer
Downsamples each feature map, usually with max pooling and a 2×2 window.
Fully connected
Flattens the final feature maps into a vector and applies one or two dense layers.
Softmax output
Turns logits into class probabilities. Trained with categorical cross-entropy.
Recurrent networks for sequences
A recurrent network processes a sequence one element at a time, carrying a hidden state from one step to the next. Past inputs influence future outputs through that state — this is the network's memory.
At each step \(t\), the hidden state \(h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)\) and the output \(y_t = W_{hy} h_t + b_y\). The cyclic dependence \(h_t \leftarrow h_{t-1}\) makes the computation graph cyclic rather than feedforward.
Unrolling over time
For analysis and training, the RNN is "unrolled" along the sequence so each time-step is a copy of the same cell with shared parameters \(W_{xh}, W_{hh}, W_{hy}\). With 3 time-steps you get 3 instances of the cell — but only one set of weights.
Backpropagation through time (BPTT)
Apply standard backprop to the unrolled graph. Gradients accumulate over time steps, and the shared weights receive the sum of contributions from every step where they appeared.
Use cases
Sentences (one word after another), audio, time series, character prediction. The lecture's worked example predicts the next character given the previous characters.
Vanishing / exploding
Repeated multiplication through many time-steps shrinks or blows up the gradient. Simple RNNs struggle with long-range dependencies for this reason.
Why LSTM
LSTM introduces gated memory cells that can hold information for many time-steps without saturation, mitigating vanishing gradients.
LSTM cells: gated long-term memory
The LSTM cell keeps a cell state \(C_t\) that runs along the sequence, and three gates that decide what to forget, what to add, and what to output. Each gate is a sigmoid producing values in \([0,1]\) used as element-wise multipliers.
Forget gate
\(f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)\). Decides what fraction of the previous cell state to keep. Output close to 0 means "forget"; close to 1 means "keep".
Input gate
\(i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)\) decides which entries to update, paired with a candidate \(\tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C)\) that proposes the new content.
Cell-state update
\(C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t\). Old state attenuated by forget, plus new candidate scaled by input gate.
Output gate
\(o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)\), then \(h_t = o_t \odot \tanh(C_t)\). The hidden state is a filtered, squashed view of the cell state.
Study questions to answer before moving on
An input image is \(28 \times 28\). A convolutional layer with filter \(5 \times 5\), stride 1, and no padding is applied. What is the output feature-map size?
Output dimension \(= (W - F)/S + 1 = (28 - 5)/1 + 1 = 24\). So the output is \(24 \times 24\).
With padding 2 and the same filter and stride, the output would stay at \(28 \times 28\) ("same" padding).
Why does an RNN that processes a sentence of length \(T\) still have only one set of weights, despite being unrolled into \(T\) time-step copies?
Because the unrolled diagram repeats the same cell at each time-step — the parameters \(W_{xh}, W_{hh}, W_{hy}\) are shared. During BPTT, each parameter receives the sum of gradient contributions from every time-step.
Explain how an LSTM can remember information from time-step 1 at time-step 50 even when a vanilla RNN cannot.
The cell-state pathway \(C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t\) lets information flow with multiplicative gates near 1, so the gradient does not necessarily shrink each step. A vanilla RNN passes everything through \(\tanh\) and a dense matrix, which causes repeated saturation and vanishing gradient.
Chapter quizzes
Self-test and math questions for this week are in the Quiz Hub (practice or exam mode).