Lecture 1: Introduction & Representing Text

Unit & tasks

What this unit is about

• NLP = algorithms mapping language ↔ computation (e.g. parse utterance → show_directions(from: …, to: …)).
• Four skills the course stresses: build, evaluate, understand, integrate NLP technology.
• Concept map: data · learning methods · count-based models · LLMs (evaluation everywhere).
• Task families: sequence tagging (POS, NER); classification; structured prediction (parsing, coreference); generation / transduction (MT, summarisation, dialogue).

RecallWithout looking up: what four verbs describe what you should be able to do with NLP tech? Name two task families from the list.

Symbolic resources

Lexicons: definitions and relations

• A good word representation should encode (when needed): morphology (e.g. -ed), animacy, number, and similarity of meaning (eat ≈ consume).
• Dictionary sense (J&M Appendix G): word = its definition — readable, but circular and limited coverage at scale.
• WordNet (Princeton): relations — hypernym/hyponym, synonym sets. English ≈ \(10^5\) nouns + large verb/adj/adv sets; 200+ languages, often smaller.
• Meronyms (part-of) also in WordNet: e.g. wheel → rim (meronym), gear (hyponym).

RecallWhat two symbolic resources are contrasted? What does WordNet use instead of definitions alone?

Sparse vectors

One-hot words and bag-of-words documents

• Map types to \(1\ldots|V|\). One-hot: single 1 at index; store as dense, sparse, set, or hash (collisions possible).
• Document vectors: OR of one-hots → binary presence; sum of counts → usual BoW (keeps frequencies).
• Tradeoff: dense length-\(|V|\) vs sparse; compare time often \(O(|V|)\) or \(O(\text{nonzeros})\).
• Limit: one-hot dims orthogonal → hall & room can have zero overlap despite semantics → need distributional / learned dense vectors.

RecallWhy are hall and room orthogonal in one-hot space? OR vs sum of one-hots — which loses counts?

Corpus weighting

TF–IDF & BM25

• \(\mathrm{count}(t,d)\) = count in doc \(d\); \(N\) = #docs; \(\mathrm{df}(t)\) = #docs containing \(t\).
• High-frequency words like the carry little document-specific signal — downweight with TF/IDF.
• Many TF–IDF variants; slides use the formulas below.
• BM25 (IR): same spirit; score = sum over query terms, IDF-like × length-normalised TF; parameters \(a \in [0.1,4]\), \(b \in [0.1,1]\) (full score on handout).

RecallWrite the IDF formula. If \(\mathrm{df}(t)=N\), what is \(\mathrm{idf}_t\)? What does BM25 add beyond TF–IDF (two words)?

Context statistics

Distributional similarity

Firth: company it keeps · Wittgenstein: meaning in use · Harris: distributional structure.

• Idea 1: per word \(w\), count co-occurrences in same sentence/window → long sparse context vector · car/automobile more similar than car/chocolate.
• Concrete example: car and automobile share context words (drove, door, luxury, speed, crashed, insurance) while chocolate does not — visualised as context-word bar charts.
• Idea 2: SVD → shorter dense vectors (Schütze 1993; pre-dates embedding boom).

RecallState the distributional hypothesis in one sentence. What does SVD do to context count vectors?

CBOW / word2vec

Neural word vectors (preview)

• Define a prediction task on raw text (e.g. predict masked word from context); cheap data + labels.
• Pipeline: context embeddings → mean → matrix × vocab → softmax → cross-entropy vs one-hot \(\mathbf{y}\) → backprop (later lectures).
• word2vec CBOW (Mikolov 2013): “continuous” = real vectors, not integer BoW counts; context e.g. ±4 words.
• Skip-gram: centre → predict context; often stronger, less intuitive.
CBOW: context words → embeddings → mean → vocabulary projection → softmax → predict centre word.
• Window: narrow → more syntactic; wide → more semantic.
• Properties (animacy, etc.) may appear as an axis, direction, or region — not always one dimension.
• Why prediction works: next-word prediction requires diverse knowledge — numerical patterns, world state, world knowledge, logic, pop culture — so the model implicitly learns language structure.

RecallList the six training steps (context → loss). CBOW vs skip-gram: who predicts whom? Narrow vs wide window?

Intrinsic tests

Evaluating vectors

• Similarity: dot product favours length; cosine \(\frac{\mathbf{u}\cdot\mathbf{v}}{|\mathbf{u}||\mathbf{v}|}\) for direction.
• Analogies: build sets (capital, element, morphology); check \(\mathbf{v}_{w_1}-\mathbf{v}_{w_2}+\mathbf{v}_{w_3}\) nearest neighbour (e.g. Paris−France+Italy ≈ Rome).
• Caveats: works best on frequent words; not all relations linear; noise in small distances.
• Bias: news-trained embeddings can encode stereotypes (Bolukbasi et al. 2016; debiasing methods exist).
• Instability: WKM 2018 — same data/model, different random seeds → different neighbours (especially rare words).

RecallWhen is cosine preferred to dot product? Write the analogy arithmetic check. Name two caveats (bias / instability / frequency).

End-to-end

Typical pipeline

1. Collect / normalise text (encoding, strip HTML).
2. Tokenise (words/subwords); optional sentence split / lemmatise.
3. Represent tokens (counts, TF–IDF, ids, embeddings).
4. Task model (classifier, tagger, LM, parser, generator).
5. Decode to labels, trees, or strings.

Pretrained transformers often merge steps; stages still help debug and baselines.

RecallRecite the five pipeline stages in order.

Lab

Workshop preview

• Pre-work: submit code practice.
• In class: explore embeddings together; discuss embedding bias.
• Later chapters: Materials/Workshop* notebooks. Grades / policies on Canvas.

Quick practice