← Study Guide Mind Map Home

Math Foundations — Lecture 5

Embedding objectives (CBOW, skip-gram, GloVe), seq2seq conditioning, F-beta, BLEU, BPE algorithm.

Dot product & similarity

\[\mathbf{u}\cdot\mathbf{v}=\sum_i u_i v_i = \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta\]

Dot product mixes magnitude and direction. Cosine similarity (\(\cos\theta\)) isolates direction. Both appear throughout embedding and attention math.

word2vec CBOW objective

Predict centre word \(w_c\) from averaged context embeddings:

\[\bar{\mathbf{h}}=\frac{1}{2m}\sum_{j\in\text{context}}\mathbf{e}_{w_j}\] \[p(w_c\mid\text{ctx})=\frac{\exp(\mathbf{u}_{w_c}^\top\bar{\mathbf{h}})}{\sum_{w'\in V}\exp(\mathbf{u}_{w'}^\top\bar{\mathbf{h}})}\]

Maximise \(\sum\log p(w_c\mid\text{ctx})\) over the corpus. Full softmax over \(|V|\) is expensive → negative sampling approximation.

Skip-gram objective

Reverse direction: predict each context word from the centre word:

\[p(w_j\mid w_c)=\frac{\exp(\mathbf{u}_{w_j}^\top\mathbf{e}_{w_c})}{\sum_{w'\in V}\exp(\mathbf{u}_{w'}^\top\mathbf{e}_{w_c})}\]

Maximise \(\sum_{c}\sum_{j\in\text{ctx}(c)}\log p(w_j\mid w_c)\). Often stronger for rare words than CBOW.

Negative sampling

\[\mathcal{L}_\text{NS}=-\log\sigma(\mathbf{u}_{w_j}^\top\mathbf{e}_{w_c})-\sum_{k=1}^{K}\mathbb{E}_{w_k\sim P_n}\bigl[\log\sigma(-\mathbf{u}_{w_k}^\top\mathbf{e}_{w_c})\bigr]\]

\(K\) negative samples drawn from noise distribution \(P_n\propto f(w)^{3/4}\).

GloVe objective

\[\mathcal{L}=\sum_{i,j=1}^{|V|}f(X_{ij})\bigl(\mathbf{w}_i^\top\tilde{\mathbf{w}}_j+b_i+\tilde b_j-\log X_{ij}\bigr)^2\]
\(X_{ij}\)Co-occurrence count of words \(i,j\) in a window.
\(f(x)\)Weighting: caps influence of very frequent pairs.

Learns vectors so that dot product approximates log co-occurrence.

Seq2seq conditioning

Encoder produces context \(\mathbf{c}\) (e.g. last hidden state). Decoder generates:

\[p(\mathbf{y}\mid\mathbf{x})=\prod_{t=1}^{T_y}p(y_t\mid y_1,\dots,y_{t-1},\mathbf{c})\]

Teacher forcing: at training, condition on gold \(y_{

Quick Check: What is the bottleneck in vanilla encoder–decoder?
The entire source compressed into one vector \(\mathbf{c}\)
The decoder's hidden size

\(F_\beta\) score (chrF)

\[F_\beta = (1+\beta^2)\frac{P\cdot R}{\beta^2 P + R}\]

\(\beta=1\) → equal weight (standard F1). \(\beta>1\) → recall-weighted. \(\beta<1\) → precision-weighted. chrF uses character n-gram P/R in this formula.

BLEU score

Geometric mean of n-gram precisions with a brevity penalty:

\[\text{BLEU}=\text{BP}\cdot\exp\!\left(\sum_{n=1}^{N}\frac{1}{N}\log p_n\right)\]
\[\text{BP}=\begin{cases}1&\text{if }c>r\\ e^{1-r/c}&\text{if }c\le r\end{cases}\]
\(p_n\)Modified n-gram precision (clipped counts).
\(c\)Candidate length.
\(r\)Effective reference length.

Brevity penalty discourages very short translations that trivially achieve high precision.

Byte-Pair Encoding (BPE)

  1. Start: vocabulary = all individual characters (+ end-of-word marker).
  2. Count all adjacent symbol pairs in the training corpus.
  3. Merge the most frequent pair into a new symbol.
  4. Repeat 2–3 until vocabulary reaches target size \(K\).

At inference: apply merges in learned order. Rare words decompose into known subwords; frequent words stay whole.

Quick Check: BPE merge criterion is:
Most frequent adjacent pair
Longest match
Highest mutual information