Lecture 8: Data — Sources, Annotation, Crowdsourcing

Where data comes from

Sources of NLP data

• Web crawl: Common Crawl (~250B pages over 17 years), FineWeb derives a cleaned subset by varying filters and measuring downstream model accuracy.
• Curated corpora: Wikipedia, news, scientific papers, books — smaller but higher signal-to-noise. "Textbooks Are All You Need" (2023) argues curated data can beat scale.
• Specialised text: code (StackOverflow, GitHub), social media, parliamentary proceedings (Canadian Hansards, European Parliament, UN — naturally parallel for translation).
• Annotated datasets (NER, coreference, parsing, sentiment) usually require explicit human labelling — not free off the web.
• Dataset choices have huge downstream impact: BERT used 16 GB of books, RoBERTa scaled to 160 GB and gained accuracy mainly from more data.

RecallName two trade-offs between web-scale crawl data and curated high-quality data.

Pre-processing

Quality, filtering, and cleaning

• Language ID — keep only documents in the target language (fastText, CLD3).
• Deduplication — exact and near-duplicate (MinHash/LSH) document removal; duplicates inflate loss curves and memorisation.
• Quality filters — heuristics (sentence length, symbol-to-word ratio, repetition) or model-based scorers (caveat: a bad scorer bakes in its bias).
• PII & toxicity — remove personal data, hate speech, NSFW boilerplate.
• FineWeb innovation: choose filters by training 1.8B-param models with each filter on/off and measuring task accuracy — empirical, not hand-wavy.

RecallWhy deduplicate before training, and what does FineWeb do differently from naive heuristic filtering?

Six properties

What makes a good dataset?

• Validity — models that perform well on the task perform well in the real world (no "Clever Hans" shortcuts).
• Reliability — labels are accurate and consistent.
• Statistical power — enough examples, including hard ones, to discriminate between models.
• Testable — clear evaluation protocol with a held-out test set.
• Publicly licensed — others can reproduce and build on the work.
• Avoids social bias — represents people fairly across gender, ethnicity, dialect, geography.

RecallWhat is a "Clever Hans" shortcut, and how would you probe for one in an NLI dataset?

Pipeline

Annotation pipeline

• 1. Collect data (previous section).
• 2. Annotation guidelines — simple, clear, concise, consistent, generalisable; include common and rare cases. Refine over time.
• 3. Pilot annotation — small sample, then compare, discuss, update the guidelines.
• 4. Train annotators — read instructions → annotate → compare with gold → discuss → re-train.
• 5. Full annotation — training data usually one label per example; test data often multiple labels + adjudication.
• 6. Quality control — attention checks (e.g. "ignore the rest of this paragraph") and consistency checks (re-do a fraction with a different annotator).
• 7. Adjudication — filter to disagreements, look together, assign a final label.
• 8. Release — choose a hosting platform (GitHub, HuggingFace Datasets) and a licence.

RecallWhy is pilot annotation done before full annotation? What does adjudication produce?

Gold standards

Linguistic resources & landmark datasets

• Penn Treebank — syntactic constituency & POS annotation; ~11,000 citations, defined the field for decades.
• OntoNotes — coreference, NER, sense tags across genres (~2,000 citations across two papers).
• WordNet — hand-built lexical database: senses, synsets, hypernym/hyponym relations.
• SQuAD — extractive question answering on Wikipedia (~11,000 citations).
• Stanford Sentiment Treebank — parse-tree-level sentiment (~11,000 citations).
• GLUE / SuperGLUE — benchmark suites bundling many tasks (CoLA, SST-2, MRPC, STS-B, QQP, MultiNLI, QNLI, RTE, WNLI) → evaluate on many tasks at once to avoid overfitting to a single benchmark.

RecallWhat is the purpose of a benchmark suite like GLUE, compared to a single dataset?

How consistent?

Inter-annotator agreement

• Cohen's kappa (two annotators, categorical): \(\kappa = \dfrac{p_o - p_e}{1 - p_e}\) — \(p_o\) is observed agreement, \(p_e\) is chance agreement.
• Fleiss' kappa — generalises Cohen's to \(>2\) annotators per item.
• Krippendorff's alpha — handles ordinal, interval, ratio data and missing values.
• Raw agreement \(p_o\) can be misleading when classes are imbalanced — high \(p_o\) but \(\kappa\) near 0 means annotators are just guessing the majority class.
• Consistent \(\neq\) correct \(\neq\) useful — two annotators can agree on a wrong label, and reliable labels can still encode a biased schema.
• If \(\kappa\) is low, investigate: unclear guidelines? careless annotators? genuinely ambiguous cases? subjective task?

RecallWhy subtract \(p_e\) instead of just reporting \(p_o\)? Give an imbalanced-class scenario where \(p_o\) is high but \(\kappa\) is low.

Many hands

Crowdsourcing

• Microwork — paid small tasks: Amazon Mechanical Turk, Prolific, Scale AI. Scales fast but raises wage and exploitation concerns.
• Games with a purpose — unpaid, fun: ESP Game (image labels), Foldit (protein folding).
• Prize competitions — Kaggle, ImageNet Challenge — money for top performers, free labour for the long tail.
• Citizen science / collaboration — Wikipedia, OpenStreetMap — unpaid, satisfying.
• Implicit work — reCAPTCHA labels scanned book text and street numbers as a side effect.
• Qualifications: approval rate (e.g. 95%+), prior HITs approved (100 / 1,000 / 5,000 / 10,000), country, custom screening tasks — control quality before paying.
• Wisdom of crowds: Galton's 1906 ox-weighing fair — 800 guesses, mean 1197 lb, true 1198 lb. Aggregation can beat any individual.

RecallList two qualifications you would require for an MTurk NER annotation HIT, and explain why.

Ethics

Risks, ethics, and documentation

• Social bias — OntoNotes: 80% of gendered pronouns are male; male mentions twice as likely to contain a job title (Zhao et al. 2018). Models learn and amplify this.
• Shortcuts & validity — annotators leak stylistic patterns; hard cases must be constructed deliberately (minimal edits, distractors with high word overlap).
• Statistical power — benchmarks where humans and models saturate quickly stop being informative.
• Tasks that shouldn't exist — predicting "criminality" from a face has no scientific basis and serious ethical concerns; refuse to build it.
• Privacy & consent — anonymise PII; obtain consent; comply with GDPR / Australian Privacy Principles.
• Copyright & licensing — confirm you have the right to redistribute; pick a licence (CC-BY, CC0, MIT, custom).
• Human subjects — annotation is human subjects research → university IRB / ethics approval; company guidelines for industry.
• Documentation: Datasheets for Datasets (Gebru et al.) and Model Cards (Mitchell et al.) record collection process, demographics, intended use, and known limitations.

RecallName two items a datasheet should record. Why is annotation considered human subjects research?

Lab

Workshop preview & materials

• Pre-work: set up two annotation tools locally (e.g. doccano and brat or Label Studio).
• In-class: annotate a small dataset with each tool, compare ergonomics; valuable preparation for Assignment 4.
• Lecture 8 handout (PDF) — full slide deck.

Quick practice