Lecture 2: Foundations of NLP Systems

Architecture

Five core components

• Data — phenomena you want the system to cover.
• Model — scores (input, output) pairs.
• Inference — pick output given input + model.
• Learning — update model from data + metric/loss.
• Metric — score prediction vs gold.
• Inference algorithms: Greedy search, A* search, Dynamic Programming — strategies when exhaustive search is intractable.
• Rule sentiment: rules + fire rules + human edits + 0/1.
• Linear: weighted features · argmax/sign · perceptron / logistic · 0-1 or CE.
• Neural: preprocess → differentiable model · gradients on CE etc.

RecallList the five components. In one line each: what does inference do? What does learning do?

Running example

Sentiment (Pang et al. 2002)

• Movie reviews → pos/neg; same template: spam, authorship, legal responsiveness, demographics from text.
• Consider ethics and consent for sensitive attributes.

RecallName two classification tasks beyond sentiment from the lecture list.

Scale

How much data?

• LM: \(10^6\)–\(10^{13}+\) words (illustrative).
• MT: \(\sim 6\times 10^7\) words × many langs.
• Sentiment: \(\sim 10^4\) sentences; syntax: \(\sim 4\times 10^4\) sentences.
• Scale ∝ task hardness & model class.

RecallOrder-of-magnitude: sentiment vs LM corpus (which is larger?).

Features

Tokenisation & BoW

• Baseline tokenisation: whitespace split — OK English; fails w/o spaces; long compounds (German).
• BoW: count vector over types; sparse dict vs dense \(|V|\) — short docs ≈ few nonzeros.
• Limits: sense conflation (bank); UNK/ drop unknowns at test.
• Stopwords: often stripped in IR; sometimes critical (authorship).
• Extras: bigrams; NOT_ negation scope (Pang); lexicon features (e.g. city list).

RecallWhy is sparse storage cheaper than full \(|V|\) for a short sentence? Name two feature tricks beyond unigrams.

Scoring

Linear classifier & multi-class

• Score = \(\sum_f \mathrm{count}_f \cdot w_f\) + optional bias (feature always 1).
• Binary: threshold score or two scores.
• Multi-class: weight vector per class → \(\arg\max_\ell \mathbf{x}\!\cdot\!\mathbf{w}_\ell\) / max-index(\(\mathbf{x}W\)); small \(|\mathcal{L}|\) → exhaustive (assignments).

\[\text{score} = \sum_{f \in \text{features}} \text{count}_f \cdot \text{weight}_{f}\]

RecallHow do you predict multi-class with linear weights? Why exhaustive label scoring is OK when \(|\mathcal{L}|\) is small?

Objectives

Learning & losses

• 0–1 · hinge \(\max(0,1-yf(x))\) · squared error · cross-entropy \(-\log p(y)\).
• Perceptron: wrong class → add \(\mathbf{x}\) to true weights, subtract from predicted.
• Logistic gradient: \((\sigma(\mathbf{x}\!\cdot\!\mathbf{w})-y)\,x_j\) per coordinate.
• Naive Bayes: estimate weight \(w_{f,c}=\frac{\text{count}(f \text{ in class }c)}{\text{count}(\text{all features in class }c)}\) from data frequencies; fast BoW baseline.

RecallPerceptron update rule in words. Logistic loss gradient form.

Dense inputs

Word vectors in the baseline

• Document vector = pool word embeddings (e.g. mean) → same linear layer: \(\mathrm{scores}=\mathrm{emb(doc)}\cdot W\).
• Pros: reuse GloVe etc.; semantics; small/fast.
• Cons: less interpretable; bias from pretraining data.

RecallOne pro and one con of embedding-based features vs raw counts.

Evaluation

Splits & metrics

• Train fit · dev/val tune (don’t train final weights on it) · test rare, no daily peeking.
• Accuracy = \(\frac{TP+TN}{\text{all}}\); Precision = \(\frac{TP}{TP+FP}\); Recall = \(\frac{TP}{TP+FN}\); F1 = \(\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN}\).
• Multi-class: per-class P/R; macro = average of class scores; micro = pool counts then P/R (micro P=R when no “none” class in slide example).
• Worked example: Spam detection with TP=4, FP=3, FN=6, TN=7 → Accuracy=0.55, Precision=4/7≈0.57, Recall=4/10=0.40, F1≈0.47.
• ROC/AUC: TPR vs FPR tradeoff; PR curve for rare positives; pick metric matching costs (spam FP costly).

RecallWrite Precision and Recall. When does micro-averaging P equal micro R in the slide example?

Lab

Workshop 2

• Pre-work: classifier pipeline (e.g. 20 Newsgroups style).
• Workshop: Diplomacy deception detection · JSONL · sklearn CountVectorizer → TfidfTransformer → LogisticRegression.
• Path: chapters/chapter2/Materials/Workshop2/ — workshop2-prework.ipynb, workshop2ED.ipynb, mod-*.jsonl.

RecallName the three pipeline stages in order.

Quick practice