Math Foundations — Lecture 8

Inter-annotator agreement (Cohen's & Fleiss' kappa, Krippendorff's alpha), observed vs chance agreement, worked examples on imbalanced and balanced classes.

Observed vs chance agreement

Suppose two annotators label \(N\) items into \(K\) categories. The most basic statistic is observed agreement:

\[p_o \;=\; \frac{\#\text{items where annotators agree}}{N} \;=\; \sum_{c=1}^{K} P(A_1 = A_2 = c)\]

But raw \(p_o\) is misleading when one class dominates. If 95% of emails are "not spam" and both annotators just always pick "not spam", they agree 95% of the time without contributing any real labelling skill. We need to correct for chance.

Expected agreement by chance is computed assuming each annotator labels independently according to their own marginal distribution:

\[p_e \;=\; \sum_{c=1}^{K} P(A_1 = c)\,P(A_2 = c)\]

Cohen's kappa

Cohen's \(\kappa\) measures agreement above chance, normalised so that perfect agreement gives \(\kappa = 1\) and chance gives \(\kappa = 0\):

\[\boxed{\;\kappa \;=\; \frac{p_o - p_e}{1 - p_e}\;}\]

\(p_o\)Observed agreement — proportion of items both annotators label the same.

\(p_e\)Expected (chance) agreement under independent marginals.

\(\kappa = 1\)Perfect agreement.

\(\kappa = 0\)Agreement no better than chance.

\(\kappa < 0\)Worse than chance — annotators systematically disagree.

Rough guide (Landis & Koch 1977): \(<0\) poor · \(0.0\text{–}0.2\) slight · \(0.2\text{–}0.4\) fair · \(0.4\text{–}0.6\) moderate · \(0.6\text{–}0.8\) substantial · \(0.8\text{–}1.0\) almost perfect. These thresholds are conventional, not statistically motivated.

Worked example 1 — imbalanced classes

Confusion matrix for 100 items, two annotators, two labels \(A, B\):

\[\begin{array}{c|cc} & A_1{=}A & A_1{=}B \\\hline A_2{=}A & 7 & 4 \\ A_2{=}B & 8 & 81 \end{array}\]

Observed agreement:

\[p_o = \frac{7 + 81}{100} = 0.07 + 0.81 = 0.88\]

Marginal probabilities:

\[P(A_1 = A) = \frac{7+8}{100} = 0.15,\quad P(A_2 = A) = \frac{7+4}{100} = 0.11\] \[P(A_1 = B) = 0.85,\quad P(A_2 = B) = 0.89\]

Chance agreement:

\[p_e = 0.15 \times 0.11 + 0.85 \times 0.89 = 0.0165 + 0.7565 = 0.773\]

Cohen's kappa:

\[\kappa = \frac{0.88 - 0.773}{1 - 0.773} = \frac{0.107}{0.227} \approx 0.471\]

Despite 88% raw agreement, \(\kappa \approx 0.47\) — only moderate, because the class imbalance inflates \(p_e\).

Worked example 2 — balanced classes

Same observed agreement (98%) but balanced marginals:

\[\begin{array}{c|cc} & A_1{=}A & A_1{=}B \\\hline A_2{=}A & 49 & 1 \\ A_2{=}B & 1 & 49 \end{array}\]

\[p_o = 0.49 + 0.49 = 0.98,\quad p_e = 0.5 \times 0.5 + 0.5 \times 0.5 = 0.5\] \[\kappa = \frac{0.98 - 0.5}{1 - 0.5} = 0.96\]

Same \(p_o\), but \(\kappa\) is far higher because chance agreement is much lower. Takeaway: always report \(\kappa\), not just raw agreement, especially when classes are unbalanced.

Quick Check: Two annotators label 100 items with two equally frequent classes. They agree on 50 items (\(p_o = 0.5\)). The marginals are 50/50 for both, so \(p_e = 0.5\). What is \(\kappa\)?

\(\kappa = 0.5\)

\(\kappa = 0\) — agreement is at chance level

\(\kappa = 1\) — perfect agreement

\(\kappa\) is undefined

Fleiss' kappa (\(\geq 2\) annotators)

For \(N\) items, \(K\) categories, \(n\) annotators per item. Let \(n_{ij}\) be the number of annotators who assigned item \(i\) to category \(j\), so \(\sum_j n_{ij} = n\).

Per-item agreement (proportion of agreeing pairs):

\[P_i \;=\; \frac{1}{n(n-1)}\sum_{j=1}^{K} n_{ij}(n_{ij}-1)\]

Average over items, and chance agreement from class-proportion squares:

\[\bar{P}_o = \frac{1}{N}\sum_{i=1}^N P_i,\qquad \bar{P}_e = \sum_{j=1}^{K}\!\left(\frac{1}{Nn}\sum_{i=1}^N n_{ij}\right)^{\!2}\]

\[\kappa_F \;=\; \frac{\bar{P}_o - \bar{P}_e}{1 - \bar{P}_e}\]

Reduces to Cohen's kappa when \(n=2\) and annotator identities are ignored.

Krippendorff's alpha

A more general coefficient that supports nominal, ordinal, interval, and ratio data, and handles missing values:

\[\alpha \;=\; 1 - \frac{D_o}{D_e}\]

\(D_o\)Observed disagreement (sum of pairwise distances within items, weighted by a distance metric chosen for the data type).

\(D_e\)Expected disagreement if labels were randomly permuted across items.

For nominal data with two annotators and no missing values, \(\alpha \approx \kappa\). For ordinal labels (e.g. Likert ratings) Krippendorff's distance function gives partial credit for being "close", which kappa does not.

If agreement is low: diagnose, don't average

Unclear guidelines — annotators interpret edge cases differently. Fix: add worked examples for the disputed cases.
Careless annotators — failed attention checks, fast clicks. Fix: block them, requalify, or use redundancy.
Ambiguous items — genuine label uncertainty in the data. Fix: flag and adjudicate, or accept multiple gold labels.
Subjective task — sentiment, offence, humour. Accept that there is no single ground truth; report distribution of labels.

Annotation is iterative: low kappa is information about your schema and annotators, not a final verdict on the data.