COMP5318 — Applied Machine Learning
Week 4: Naive Bayes & Evaluation
Week 4 combines a probabilistic classifier with the evaluation discipline needed for the rest of the course. The lecture covers Bayes' theorem, Naive Bayes, holdout, cross-validation, leave-one-out, stratification, and grid search; the notebook shows the sklearn workflow on iris; the tutorial solutions make you compute posteriors by hand for both nominal and numeric features.
Course materials
This is the week where "getting a good score once" stops being enough. The source materials force you to separate model building, parameter tuning, and final reporting.
| File | What to extract from it |
|---|---|
| w4.ipynb | Sklearn usage for GaussianNB, precision/recall/F1, confusion matrices, cross-validation, leave-one-out, and grid search. |
| w4-solutions.ipynb | Interpretation of per-class metrics, reliability of cross-validation, and the role of the training set inside grid search. |
| ml4.pdf | Bayes theorem, Naive Bayes assumptions, missing values, Gaussian treatment of numeric features, evaluation procedures, and performance measures. |
| t4.pdf | Manual Naive Bayes on loan-default data for nominal and numeric attributes. |
| t4-solutions.pdf | Worked posterior calculations, Gaussian likelihood values, class priors, and final decisions. |
Naive Bayes from lecture formula to sklearn model
The lecture introduces probabilistic classifiers as models that estimate class-membership probabilities. Naive Bayes is the simplest prominent example: it is easy to compute because it assumes the features are conditionally independent given the class.
\[ P(H \mid E) = \frac{P(E \mid H)P(H)}{P(E)} \] In classification terms, \(H\) is a class hypothesis and \(E\) is the observed feature vector.
\[ P(E \mid H) = \prod_i P(E_i \mid H) \] The "naive" part is the conditional-independence assumption. The lecture explicitly says this assumption is unrealistic but often still works surprisingly well.
Classifier variants in the notebook
The notebook lists four sklearn variants: GaussianNB, CategoricalNB, MultinomialNB, and BernoulliNB. For Week 4, iris is numeric, so the notebook uses GaussianNB.
Numeric features
The lecture handles numeric attributes by assuming a Gaussian likelihood per class. The tutorial's second exercise follows this exactly: estimate class-specific mean and standard deviation, then evaluate the Gaussian density at the new value.
Robustness
The lecture notes that Naive Bayes is robust to isolated noisy points because one unusual example has only limited effect on conditional probabilities.
Weak spot
Correlated features reduce its power because correlation violates the independence assumption. The lecture suggests feature selection as a remedy.
Missing values
The lecture points out that Naive Bayes can deal with missing values naturally by skipping unavailable attribute terms during classification.
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
iris.data, iris.target, stratify=iris.target, random_state=42
)
nb = GaussianNB()
nb.fit(X_train, y_train)
y_pred = nb.predict(X_test)Beyond accuracy: confusion matrices, precision, recall, and F1
The notebook and lecture both push the same message: accuracy alone can hide important failure modes. You need a confusion matrix and class-sensitive metrics to see where errors go.
Confusion matrix
Rows are true classes and columns are predicted classes. Correct classifications lie on the main diagonal.
Precision
Of the examples predicted as a class, how many were actually that class?
Recall
Of the examples truly in a class, how many did the model recover?
F1 score
The harmonic mean of precision and recall, useful when both matter and class imbalance exists.
Notebook interpretation for iris: the solutions notebook emphasises that precision, recall, and F1 are reported per class, plus averages. It also explains how to read a multi-class confusion matrix: for its iris run, 1 example of class 1 was misclassified as class 2 and 2 examples of class 2 were misclassified as class 1.
from sklearn import metrics
actual = y_test
predicted = nb.predict(X_test)
print(metrics.classification_report(actual, predicted))
print(metrics.confusion_matrix(actual, predicted))Holdout, cross-validation, leave-one-out, and grid search
The evaluation half of Week 4 is foundational. It tells you which score you are allowed to trust, what the validation loop is doing, and why the test set must remain untouched until the end.
Holdout
Split once into train and test. Easy, but variance can be high because the result depends on a single random split.
Stratification
Preserve class proportions in the splits so rare classes are not accidentally missing from training or test data.
10-fold CV
The lecture calls stratified 10-fold cross-validation the standard evaluation method for classification in ML.
LOOCV
Use each example as its own test fold once. Better use of data, but slow because the number of evaluations equals the number of examples.
When hyperparameters must be tuned, the lecture explicitly separates training, validation, and test. The test set cannot be used for parameter tuning. Grid search with cross-validation operationalises exactly this idea.
from sklearn.model_selection import cross_val_score, LeaveOneOut, GridSearchCV
from sklearn.neighbors import KNeighborsClassifier
scores = cross_val_score(nb, iris.data, iris.target, cv=10)
one_out = LeaveOneOut()
loo_scores = cross_val_score(nb, iris.data, iris.target, cv=one_out)
param_grid = {"n_neighbors": [1, 3, 5, 11, 15], "p": [1, 2]}
grid_search = GridSearchCV(KNeighborsClassifier(), param_grid, cv=10, return_train_score=True)
grid_search.fit(X_train, y_train)grid_search.best_score_ is the mean cross-validation score computed inside the training data. grid_search.score(X_test, y_test) is the final held-out test score of the model retrained on the full training set with the chosen parameters. These are not the same number and they serve different purposes.
Tutorial calculations worth memorising structurally
The new example is home owner = no, marital status = married, annual income = very high. The solutions compute both class posteriors and conclude:
\(P(no \mid E) > P(yes \mid E)\), so the prediction is loan default = no.
The same example is revisited with numeric income \(=120\). The solutions first estimate class-specific Gaussian parameters:
- Class yes: \(\mu=99\), \(\sigma=15.57\)
- Class no: \(\mu=109\), \(\sigma=66.18\)
Then they compute \(f(income=120\mid yes)\) and \(f(income=120\mid no)\), combine them with the nominal-feature probabilities, and again conclude loan default = no.
The important study pattern is: separate the data by class, estimate the Gaussian parameters per class, evaluate the density of the new feature value under each class, then multiply by the class prior and any remaining feature-conditionals. You compare posteriors; you do not need to compute the common denominator \(P(E)\) explicitly if you only care about the winning class.
Chapter quizzes
Self-test and math questions for this week are in the Quiz Hub (practice or exam mode).