COMP5318 — Week 4 Supplement
Mathematical Foundations
Bayes theorem, Naive Bayes factorisation, confusion-matrix metrics, and cross-validation logic for evaluating model quality instead of just quoting one accuracy number.
Bayes theorem and the Naive Bayes simplification
Week 4 moves from geometric decision boundaries to probabilistic classification. The mathematical question becomes: given the observed feature values \(E\), which class \(c\) makes those observations most plausible?
Bayes theorem
\[ P(H\mid E)=\frac{P(E\mid H)P(H)}{P(E)} \]
Prior \(P(H)\): belief in class \(H\) before seeing the example.
Likelihood \(P(E\mid H)\): how compatible the evidence is with class \(H\).
Posterior \(P(H\mid E)\): updated class probability after seeing the evidence.
For attributes \(E_1,\dots,E_d\), the independence assumption gives
\[ P(c\mid E_1,\dots,E_d)\propto P(c)\prod_{j=1}^d P(E_j\mid c) \]
The denominator \(P(E)\) is the same for every class, so for classification we usually compare only the numerators.
Numeric features in Gaussian NB
When a feature is numeric, the tutorial models \(P(E_j\mid c)\) with a Gaussian density. That means estimating a class-specific mean \(\mu_c\) and standard deviation \(\sigma_c\), then evaluating
\[ f(x)=\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2\sigma^2}} \]
Worked Example 1: nominal Naive Bayes from the tutorial
Use the Week 4 tutorial loan-default dataset to classify the new example:
home owner = no, marital status = married, annual income = very high
Posterior numerator for class yes
From the table:
\(P(\text{yes})=5/10\)
\(P(E_1\mid\text{yes})=P(\text{home owner=no}\mid\text{yes})=3/5\)
\(P(E_2\mid\text{yes})=P(\text{married}\mid\text{yes})=1/5\)
\(P(E_3\mid\text{yes})=P(\text{very high}\mid\text{yes})=1/5\)
So the unnormalised posterior is
\[ \frac{3}{5}\cdot\frac{1}{5}\cdot\frac{1}{5}\cdot\frac{5}{10} =\frac{3}{250}=0.012 \]
Posterior numerator for class no
Similarly:
\(P(\text{no})=5/10\)
\(P(E_1\mid\text{no})=3/5\)
\(P(E_2\mid\text{no})=3/5\)
\(P(E_3\mid\text{no})=2/5\)
\[ \frac{3}{5}\cdot\frac{3}{5}\cdot\frac{2}{5}\cdot\frac{5}{10} =\frac{9}{125}=0.072 \]
Because \(0.072 > 0.012\), Naive Bayes predicts loan default = no.
Confusion matrix, precision, recall, and F1
When classes are imbalanced or false positives and false negatives matter differently, accuracy by itself is too blunt. The lecture introduces the confusion matrix to split performance into more interpretable pieces.
Accuracy
\[ \frac{TP+TN}{TP+FP+FN+TN} \]
Precision
\[ \frac{TP}{TP+FP} \] “Of the predicted positives, how many were correct?”
Recall
\[ \frac{TP}{TP+FN} \] “Of the real positives, how many did we recover?”
F1
\[ \frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN} \] Harmonic mean of precision and recall.
Worked Example 2: compute metrics from a confusion matrix
Suppose a binary classifier produces the following counts on a test set: \(TP=12\), \(FP=3\), \(FN=5\), \(TN=30\).
Accuracy, precision, recall
\[ \text{Accuracy}=\frac{12+30}{12+3+5+30}=\frac{42}{50}=0.84 \]
\[ \text{Precision}=\frac{12}{12+3}=\frac{12}{15}=0.80 \]
\[ \text{Recall}=\frac{12}{12+5}=\frac{12}{17}\approx0.706 \]
F1 and interpretation
\[ F1=\frac{2TP}{2TP+FP+FN}=\frac{24}{24+3+5}=\frac{24}{32}=0.75 \]
This model is reasonably precise, but recall is lower. It misses 5 true positives, which pulls the F1 score below precision.
Why cross-validation is the standard evaluation procedure
The notebook emphasizes that a single train/test split can be noisy because the result depends on which examples happened to land in the test set. Cross-validation reduces that variance by rotating the hold-out part.
10-fold cross-validation in plain language
- Split the training data into 10 folds.
- Train on 9 folds and evaluate on the remaining fold.
- Repeat until every fold has served as validation once.
- Average the 10 scores to estimate generalisation performance.
Parameter tuning rule: tune with cross-validation on the training set only, then evaluate once on the untouched test set. If you tune on the test set, information leaks and the final score stops being trustworthy.