COMP5318 — Applied Machine Learning
Week 3: Linear & Logistic Regression
This chapter moves from memory-based models to parametric models. The lecture explains linear regression, logistic regression, overfitting, and regularisation; the notebook turns that into sklearn workflows on wave data, California housing, and the breast-cancer dataset; the solutions show how Ridge, Lasso, and C tuning change behaviour in practice.
Course materials
Week 3 is one of the most reusable chapters in the unit because the same ideas reappear later: fit on training data, watch train-vs-test behaviour, and tune regularisation instead of assuming the default is right.
| File | What it teaches |
|---|---|
| w3.ipynb | Wave-data regression, California housing, extended polynomial features, Ridge, Lasso, and logistic regression in sklearn. |
| w3-solutions.ipynb | Interpretation of underfitting vs overfitting, alpha sweeps, and why one Lasso setting is especially attractive. |
| ml3.pdf | Lecture concepts: regression lines, \(R^2\), logistic curve, log-odds, overfitting, underfitting, Ridge, and Lasso. |
Main distinction
Linear regression predicts a numeric target. Logistic regression predicts a class via a probability in \((0,1)\).
Main danger
When feature space becomes richer, the model can fit the training data too closely and fail badly on new data.
Main cure
Regularisation shrinks or removes coefficients so the fitted model becomes less complex and more likely to generalise.
Linear regression and what \(R^2\) is telling you
The lecture starts with simple regression: fit a straight line that approximates the relationship between a feature \(x\) and a numeric target \(y\). The notebook extends that idea to multiple features and real datasets.
For multiple regression the prediction is linear in the features: \(\hat{y} = w^\top x + b\). Training chooses coefficients that minimise squared error on the training set.
Notebook example 1: wave data
The notebook generates a synthetic one-feature dataset with make_wave. It is deliberately simple so you can see what fitting a line means before touching a larger real dataset.
The sklearn API is minimal: create LinearRegression(), fit(X_train, y_train), then inspect coef_, intercept_, and score.
Notebook example 2: California housing
The real dataset has 20,640 examples, 8 numeric features, and a numeric housing-value target. On this version, train and test \(R^2\) are similar, so plain linear regression does not obviously overfit.
This matters because later you compare it with the expanded 44-feature version where behaviour changes dramatically.
Interpret \(R^2\) carefully: the notebook and lecture both emphasise that a good model has similar train and test performance. On the extended California example, the training \(R^2\) is good while the test \(R^2\) becomes negative, which means the fitted model performs worse on the test set than simply predicting the mean target value.
Split
Always create separate training and test subsets before fitting.
Fit
Use the training set only to estimate coefficients.
Compare scores
Similar train and test scores suggest stable generalisation; a wide gap suggests overfitting.
Question the features
Adding more features can help, but it can also create a model that is too flexible.
Regularisation: Ridge and Lasso
The lecture defines overfitting as high performance on training data but low performance on test data. Regularisation is the direct response: make the model more restrictive so it cannot chase every accidental pattern in the training set.
Ridge (L2)
\[ \frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i-y_i)^2 + \alpha\sum_{j=1}^{m} w_j^2 \]
Ridge keeps all features but shrinks coefficients toward zero. Larger alpha means a more constrained model. Very small alpha makes Ridge behave like plain linear regression.
Lasso (L1)
\[ \frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i-y_i)^2 + \alpha\sum_{j=1}^{m} |w_j| \]
Lasso not only shrinks coefficients, it can set some of them to exactly zero. That makes it a built-in feature-selection mechanism.
PolynomialFeatures(degree=2, include_bias=False), producing 44 features. Plain linear regression overfits here; regularisation is not optional, it is the point of the exercise.
Underfitting case
The solutions notebook explains that default Lasso with alpha=1 is too aggressive: both train and test scores are low and all coefficients can collapse to zero.
Best illustrated tradeoff
The provided solutions single out alpha=0.0001 as a strong Lasso setting: it beats Ridge on the test set while using only 19 of the 44 features.
Reading the trend
If alpha becomes too small, the effect of regularisation fades and Lasso starts to behave like overfitting linear regression again.
Logistic regression: linear model, probabilistic output
Despite the name, logistic regression is used for classification. The lecture's key move is to replace the straight regression line with a sigmoid that maps any real-valued score into a probability between 0 and 1.
\[ p(y=1\mid x)=\sigma(w^\top x+b)=\frac{1}{1+e^{-(w^\top x+b)}} \] The model outputs a probability for class 1. A common classification rule is to predict class 1 when \(p \ge 0.5\).
\[ \log\frac{p}{1-p} = w^\top x + b \] This is why logistic regression is still linear in the parameters even though the output probability curve is nonlinear.
Notebook dataset
The notebook uses the Breast Cancer Wisconsin dataset: 569 examples, 30 features describing cell nuclei from a biopsy, and a binary target of malignant vs benign.
The example uses LogisticRegression(solver='liblinear') after a stratified train/test split.
Regularisation parameter
For logistic regression the control knob is C, not alpha. The notebook explains that this behaves in the opposite direction to Ridge/Lasso alpha: a larger C means weaker regularisation.
High C focuses more on fitting training data; low C shrinks coefficients more strongly.
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
cancer.data, cancer.target, stratify=cancer.target, random_state=42
)
logreg = LogisticRegression(solver="liblinear")
logreg.fit(X_train, y_train)The naming trap: logistic regression is called "regression" because it models a continuous quantity, the probability or equivalently the log-odds, but the end task in this chapter is classification.
C change logistic regression regularisation in the notebook discussion?How to talk about Week 3 in an exam or assignment
If training performance is high but test performance collapses, your first interpretation should be overfitting. If both are poor, the solutions notebook explicitly labels that as underfitting.
Adding polynomial features can increase model capacity sharply. The Week 3 housing example is there to show that a richer feature space is not automatically better unless you control complexity.
The extended housing design matrix has 44 features instead of 8. That gives linear regression many more ways to fit idiosyncrasies of the training set. Ridge and Lasso work because they restrict coefficients, making the effective model smoother and less likely to chase noise.
Typical oral explanation: linear regression predicts a number directly; logistic regression predicts a probability and then thresholds it for classification; both are linear in the features before the final output transformation.
Chapter quizzes
Self-test and math questions for this week are in the Quiz Hub (practice or exam mode).