Math Foundations — Lecture 2

Linear scoring, sigmoid, losses (hinge / CE), perceptron, logistic gradient, P / R / F1, ROC.

Linear classifier scoring

Feature vector \(\mathbf{x}\in\mathbb{R}^F\), weight vector \(\mathbf{w}\in\mathbb{R}^F\), optional bias \(b\):

\[\text{score} = \mathbf{x}\cdot\mathbf{w}+b = \sum_{f=1}^{F} x_f\, w_f + b\]

Binary: threshold at 0 (or use two scores). Multi-class (\(|\mathcal{L}|\) labels): weight matrix \(W\in\mathbb{R}^{F\times|\mathcal{L}|}\), predict \(\hat y=\arg\max_\ell\;\mathbf{x}\,W_{:,\ell}\).

Sigmoid function

\[\sigma(z)=\frac{1}{1+e^{-z}}\]

Maps any real number to \((0,1)\). Derivative: \(\sigma'(z)=\sigma(z)(1-\sigma(z))\). Used in logistic regression to turn scores into probabilities.

Loss functions

0–1 loss

\[\ell_{0\text{-}1}=\mathbb{1}[\hat y \neq y]\]

Non-differentiable; cannot directly gradient-descend on it.

Hinge loss (SVM)

\[\ell_\text{hinge}=\max(0,\;1-y\,f(x))\]

where \(y\in\{-1,+1\}\) and \(f(x)\) is the raw score. Margin-based; pushes score of correct class to exceed incorrect by at least 1.

Cross-entropy / log loss

\[\ell_\text{CE}=-\log p(y\mid\mathbf{x})=-\log\frac{e^{s_y}}{\sum_\ell e^{s_\ell}}\]

where \(s_\ell=\mathbf{x}\cdot\mathbf{w}_\ell\). Smooth, differentiable; standard for neural NLP.

Quick Check: Which loss is non-differentiable and therefore hard to optimise with SGD?

Hinge

Cross-entropy

0–1 loss

Perceptron update

If prediction \(\hat y\neq y\):

\[\mathbf{w}_y \;\leftarrow\; \mathbf{w}_y + \mathbf{x}, \qquad \mathbf{w}_{\hat y} \;\leftarrow\; \mathbf{w}_{\hat y} - \mathbf{x}\]

Add the input to the true-class weights; subtract from the predicted-class weights. No learning rate. Converges if data is linearly separable.

Logistic regression gradient

Binary case, \(y\in\{0,1\}\), prediction \(\hat p=\sigma(\mathbf{x}\cdot\mathbf{w})\):

\[\frac{\partial\mathcal{L}}{\partial w_j} = (\hat p - y)\,x_j = (\sigma(\mathbf{x}\cdot\mathbf{w})-y)\,x_j\]

Gradient descent step: \(w_j\leftarrow w_j - \eta(\hat p - y)x_j\), where \(\eta\) is the learning rate.

Quick Check: If the model predicts \(\hat p=0.9\) and the true label is \(y=1\), what is the sign of the gradient for each weight?

Negative (pushes weights to increase score)

Positive (pushes weights to decrease score)

Precision, recall, F1

Confusion matrix entries: TP, FP, FN, TN.

\[\text{Precision}=\frac{TP}{TP+FP},\qquad \text{Recall}=\frac{TP}{TP+FN}\]

\[F_1=\frac{2\,P\,R}{P+R}=\frac{2\,TP}{2\,TP+FP+FN}\]

\(F_1\) is the harmonic mean — penalises extremes more than arithmetic mean.

Macro vs micro averaging

Macro: compute P/R per class, then average. Micro: pool all TP/FP/FN across classes, then compute P/R. Micro P = micro R when every instance gets exactly one label (no "none" class).

Accuracy

\[\text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}\]

Misleading with class imbalance (e.g. 99% negative → always-negative baseline = 99%).

ROC & AUC

Sweep the decision threshold; plot TPR (recall) vs FPR (\(=FP/(FP+TN)\)).

\[\text{TPR}=\frac{TP}{TP+FN},\qquad\text{FPR}=\frac{FP}{FP+TN}\]

AUC = area under the ROC curve. Random classifier → AUC = 0.5; perfect → 1.0.

When positives are rare, Precision–Recall curve is more informative than ROC.

Quick Check: A spam filter where false positives are very costly (important mail deleted). Which metric matters most?

High precision (few FP)

High recall (few FN)