COMP5318 — Applied Machine Learning
Week 6: SVMs & PCA
Week 6 combines a strong discriminative classifier with a strong unsupervised representation method. The SVM lecture explains maximum-margin hyperplanes, support vectors, soft margins, and kernels; the dimensionality-reduction lecture explains PCA, SVD, feature extraction, and compression; the notebook applies SVM and PCA to breast cancer, moons, and MNIST data.
Course materials
This is a high-concept week. The lecture slides give the geometric intuition; the notebook shows the practical sklearn moves that make those ideas useful.
| File | What you should learn from it |
|---|---|
| w6.ipynb | Breast-cancer SVMs, moons hyperparameter intuition, PCA on breast cancer, explained-variance selection, and MNIST compression tasks. |
| ml6a.pdf | Support Vector Machines: maximum-margin hyperplanes, support vectors, soft margins, nonlinear mappings, kernel trick, and common kernels. |
| ml6b.pdf | Dimensionality reduction: motivation, PCA geometry, choosing dimensions, SVD, feature extraction, and compression examples. |
Support Vector Machines: geometry first, code second
The SVM lecture is about one core idea: among separating boundaries, prefer the one with the largest margin. A larger margin usually means more robustness to small perturbations in the data and therefore better generalisation.
The separating hyperplane lies midway between two margin boundaries \(H_1\) and \(H_2\). The points that touch those boundaries are the support vectors. They are the training examples that actually determine the solution.
Hard margin
If the classes are linearly separable, the optimisation seeks the hyperplane with the largest margin and no training errors.
The lecture connects larger margins to lower sensitivity to noise and less overfitting.
Soft margin
If perfect separation is unrealistic, SVM introduces a tradeoff between margin width and classification error. The lecture uses the hyperparameter C for this.
Large C emphasises fewer training errors; smaller C allows a wider, more regularised margin.
Scaling is not optional
The notebook normalises the breast-cancer data before fitting SVMs because distance and dot-product based methods are sensitive to feature scale.
Linear vs nonlinear
The notebook compares kernel="linear", kernel="poly", and kernel="rbf". The lecture explains that nonlinear SVMs work by mapping data to a space where linear separation is easier.
Kernel trick
The lecture's key computational idea is that you can work with dot products in the original space while implicitly representing a higher-dimensional feature map.
Polynomial kernel
\(K(x,y) = (x \cdot y + 1)^p\). Useful when interactions of bounded degree are meaningful.
RBF kernel
\(K(x,y)=\exp(-\|x-y\|^2 / 2\sigma^2)\). Smooth nonlinear boundary; the notebook explores its behaviour via gamma.
Gamma
Controls the width of the RBF influence region. Small gamma gives smoother boundaries; large gamma creates more local, wiggly boundaries.
LinearSVC note
The notebook recommends trying a linear kernel first and notes that LinearSVC is often faster than SVC(kernel="linear") on large datasets.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
cancer.data, cancer.target, stratify=cancer.target, random_state=42
)
scaler = MinMaxScaler().fit(X_train)
X_train_norm = scaler.transform(X_train)
X_test_norm = scaler.transform(X_test)Notebook tuning intuition on moons: the four-way plot over \((\gamma, C)\) is there to make regularisation visible. Smaller gamma means broader influence and smoother boundaries; larger gamma means more local influence. Smaller C means a more restricted model; larger C bends harder to fit difficult points.
gamma a lot in an RBF SVM?PCA: lower-dimensional structure without labels
The dimensionality-reduction lecture begins with a practical problem: high-dimensional data is slow, sparse, noisy, and hard to interpret. PCA addresses this by replacing the original axes with new orthogonal axes ordered by captured variance.
PCA finds orthogonal axes \(Z_1, Z_2, \dots\) such that \(Z_1\) captures the largest variance, \(Z_2\) the next largest, and so on. The data is then projected onto the leading components.
Why it is unsupervised
PCA uses only the feature matrix \(X\). The lecture explicitly states that it does not use class labels, which is why PCA is an unsupervised method.
Choosing how many dimensions
The lecture gives two strategies: preserve a chosen percentage of variance, or inspect the explained-variance curve for an elbow.
Lecture example: iris
The PCA-on-iris slide reports that PC1 captures about 92.5% of the variance and PC2 about 5.3%, so a 2D representation preserves almost all variance.
Notebook example: breast cancer
The notebook reduces 30 features down to 2 components and then compares 1-NN on the original and reduced data. It reports roughly 0.94 vs 0.92 accuracy, showing that a tiny representation can still remain useful.
95% variance result
On the breast-cancer notebook example, preserving 95% of the variance requires 9 principal components.
from sklearn.decomposition import PCA
from sklearn.neighbors import KNeighborsClassifier
pca = PCA(n_components=2).fit(X_train_norm)
X_train_pca = pca.transform(X_train_norm)
X_test_pca = pca.transform(X_test_norm)
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train_pca, y_train)Compression
The lecture and notebook both treat PCA as a compression tool. On MNIST, the slides note that 784 original features can be reduced to about 153 while preserving 95% variance.
Feature extraction
The lecture also frames PCA as representation learning: if the projected features separate classes better or more compactly, a downstream classifier may become simpler or more accurate.
fit on the training data only and then transform both train and test with the learned projection.
Study questions to answer before moving on
Why does a large-margin separator usually generalise better than a tiny-margin separator?
The lecture's rationale is robustness: if the margin is small, slight perturbations of the data or decision boundary can flip classifications. A larger margin leaves more room for perturbation and is therefore less sensitive to noise and overfitting.
Why must PCA be fitted on the training set only, even though it is unsupervised?
Because the learned projection still depends on the distribution of the data. Fitting PCA on the full dataset would let information from the test distribution leak into preprocessing, just like fitting a scaler on the full dataset.
The notebook's practical advice is: start with a linear SVM, then try RBF if needed; for PCA, either target a variance threshold such as 95% or inspect the elbow of the explained-variance curve rather than choosing dimensions arbitrarily.
Chapter quizzes
Self-test and math questions for this week are in the Quiz Hub (practice or exam mode).