COMP5318 — Week 3 Supplement
Mathematical Foundations
Least-squares fitting, residuals, \(R^2\), logistic probabilities, and the regularisation terms that control overfitting in Ridge and Lasso.
Linear regression: objective, residuals, and \(R^2\)
The lecture frames regression as fitting a straight line or hyperplane that predicts a numeric target. The mathematics is about choosing coefficients that make prediction errors as small as possible in squared form.
Core regression formulas
Prediction: \(\hat{y}=b_0+b_1x\) in the simple one-feature case, or \(\hat{y}=Xw+b\) in matrix form.
Residual for example \(i\): \(\varepsilon_i=y_i-\hat{y}_i\)
Sum of squared errors: \[\operatorname{SSE}=\sum_{i=1}^n(y_i-\hat{y}_i)^2\]
Total squared variation: \[\operatorname{SST}=\sum_{i=1}^n(y_i-\bar{y})^2\]
Coefficient of determination: \[R^2=1-\frac{\operatorname{SSE}}{\operatorname{SST}}\]
Ordinary least squares solves \(\min_w \lVert Xw-y \rVert^2\). When \(X^\top X\) is invertible, the normal-equation solution is \[w=(X^\top X)^{-1}X^\top y.\]
Worked Example 1: fitting a simple regression line
Suppose the training data are \((x,y)=(1,3),(2,5),(3,7)\). The points lie exactly on a line, so this is a clean place to see the formulas.
Find slope and intercept
The means are \(\bar{x}=2\) and \(\bar{y}=5\).
\[ b_1=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x})^2} =\frac{(-1)(-2)+0\cdot0+(1)(2)}{(-1)^2+0^2+1^2} =\frac{4}{2}=2 \]
\[ b_0=\bar{y}-b_1\bar{x}=5-2\cdot2=1 \]
So the fitted line is \(\hat{y}=1+2x\).
Prediction, residual, and \(R^2\)
At \(x=4\), the prediction is \(\hat{y}=1+2(4)=9\).
For the training points, predictions are exactly \(3,5,7\), so every residual is \(0\). Therefore \(\operatorname{SSE}=0\).
Because \(R^2=1-\operatorname{SSE}/\operatorname{SST}\), we get \(R^2=1\): a perfect fit.
Logistic regression: from linear score to class probability
Logistic regression still builds a linear score, but instead of using that score directly as a numeric prediction, it passes it through a sigmoid curve so the output can be interpreted as a probability.
Core logistic formulas
Linear score: \(z=w^\top x+b\)
Sigmoid: \[\sigma(z)=\frac{1}{1+e^{-z}}\]
Probability model: \(P(y=1\mid x)=\sigma(w^\top x+b)\)
Logit form: \[\log\frac{p}{1-p}=w^\top x+b\]
Thresholding at \(0.5\) is equivalent to asking whether \(w^\top x+b\) is positive.
Worked Example 2: logistic probability and regularisation effect
Probability from a linear score
Take a one-feature model with \(w=0.8\), \(b=-1\), and a new input \(x=3\).
\[ z=0.8(3)-1=1.4 \]
\[ P(y=1\mid x)=\sigma(1.4)=\frac{1}{1+e^{-1.4}}\approx 0.80 \]
With a \(0.5\) threshold, this example is classified as the positive class.
How Ridge and Lasso change the objective
Suppose \(w=(3,-1,0.5)\) and \(\alpha=0.2\).
Ridge penalty: \[ \alpha \sum_j w_j^2 = 0.2(9+1+0.25)=2.05 \]
Lasso penalty: \[ \alpha \sum_j |w_j| = 0.2(3+1+0.5)=0.9 \]
The penalties measure complexity differently. Ridge spreads shrinkage smoothly across all weights; Lasso is more willing to set some weights exactly to zero.
Overfitting, Ridge, and Lasso
The lecture and notebook both show that adding too many effective degrees of freedom can make training performance look great while test performance collapses. Regularisation deliberately makes the model simpler to improve generalisation.
Ridge
\[ \frac{1}{n}\sum_{i=1}^n(\hat{y}_i-y_i)^2 + \alpha \sum_{j=1}^m w_j^2 \] Penalises large weights with an \(L2\) term.
Lasso
\[ \frac{1}{n}\sum_{i=1}^n(\hat{y}_i-y_i)^2 + \alpha \sum_{j=1}^m |w_j| \] Uses an \(L1\) penalty, so sparse solutions are common.
Interpretation
Higher \(\alpha\) means stronger restriction in Ridge/Lasso. For logistic regression, the tutorial parameter is \(C\), where smaller \(C\) means stronger regularisation.