Page 1 – Distributional Semantics

1.1 The Distributional Hypothesis

The foundational idea behind modern embedding techniques is the Distributional Hypothesis:

“You shall know a word by the company it keeps.” — J. R. Firth (1957)

Words that occur in similar contexts tend to have similar meanings. This idea motivates representing words by their contextual statistics.

1.2 Vector Space Models (VSM)

Classical VSMs represent words as high-dimensional vectors derived from co-occurrence counts. For example:

• Count occurrences of target word with context words.
• Form a word-by-context matrix.
• Apply weighting schemes (TF–IDF, PMI).

1.3 Pointwise Mutual Information (PMI)

PMI measures the strength of association between two words:

PMI(w, c) = log (P(w, c) / (P(w)P(c)))

High PMI indicates meaningful co-occurrence beyond chance.

1.4 Limitations of Classical Count-Based Methods

• High dimensionality (vocabulary-sized vectors).
• Sparsity—most co-occurrence counts are zero.
• No modelling of semantic similarity beyond local contexts.
• Cannot naturally handle polysemy (multiple meanings).

1.5 Rise of Distributed Representations

Neural embedding models solve these issues by learning dense vectors (e.g., 50–300 dimensions) that capture:

• Syntactic relationships
• Semantic similarity
• Analogical structure (e.g., king – man + woman ≈ queen)

Distributed representations are foundational to modern NLP.

Page 2 – Word2Vec and Neural Embeddings

2.1 Origins of Word2Vec

Word2Vec (Mikolov et al., 2013) revolutionised NLP by introducing efficient neural models for learning dense word embeddings from large corpora.

2.2 Continuous Bag-of-Words (CBOW)

CBOW predicts a target word from its surrounding context:

P(w | context)

Steps:

• Average embeddings of context words.
• Feed into softmax to predict target word.

2.3 Skip-Gram Model

Skip-Gram predicts context words from a target word:

P(context | w)

Skip-Gram is better for rare words; CBOW is faster.

2.4 Negative Sampling

Softmax over a large vocabulary is expensive. Negative sampling solves this by updating only a few “negative” terms per iteration.

2.5 Subsampling Frequent Words

Very frequent function words (the, is, and) are subsampled to reduce noise.

2.6 Training Objective

Word2Vec trains to maximise co-occurrence likelihood. This leads to vectors that encode semantic regularities and meaningful distances in embedding space.

2.7 Embedding Properties

Word2Vec embeddings exhibit:

• Linear analogies (king – man + woman ≈ queen)
• Semantic clusters (countries, colours, professions)
• Syntactic groupings (verbs, adjectives)

Page 3 – GloVe, FastText & Hybrid Models

3.1 GloVe: Global Vectors

GloVe (Pennington et al., 2014) combines advantages of count-based and predictive models. It constructs a weighted least-squares objective based on global word co-occurrence statistics.

3.2 GloVe Training Objective

The model learns embeddings such that:

wᵢ ⋅ wⱼ ≈ log(Xᵢⱼ)

where Xᵢⱼ is the number of times words i and j co-occur.

3.3 Advantages of GloVe

• Captures global statistical structure.
• Efficient with large corpora.
• Produces high-quality analogy performance.

3.4 FastText: Subword Embeddings

FastText (Bojanowski et al., 2017) extends Word2Vec by representing words as bags of character n-grams:

• "apple" → char n-grams like “app”, “ppl”, “ple”

This allows:

• Handling rare words via shared character features.
• Better performance for morphologically rich languages.
• Automatic construction of vectors for unseen words.

3.5 Common Embedding Benchmarks

• Word similarity datasets (SimLex-999, WordSim-353)
• Analogy tasks (Google analogy dataset)
• Downstream classification tasks

3.6 Limitations of Static Embeddings

• One vector per word → cannot represent polysemy.
• No contextual influence.
• Limited capacity for capturing syntax and pragmatics.

Page 4 – Contextual Embeddings

4.1 Motivation

Static embeddings assign a single vector to a word regardless of context. However, meaning often depends on surrounding words:

“bank” in “river bank” vs. “financial bank”.

4.2 ELMo

ELMo (Peters et al., 2018) introduced contextual embeddings using bidirectional LSTMs. Each occurrence of a word has a context-specific vector.

4.3 BERT and Contextual Encoding

BERT (Devlin et al., 2019) uses Transformers to generate contextual embeddings from both left and right context (bidirectional). Words acquire different vectors depending on usage.

Benefits:

• Handles polysemy
• Captures syntax & long-range dependencies
• Forms the backbone of modern NLP systems

4.4 Types of Contextual Embeddings

• Token embeddings – each token has its own vector.
• Sentence embeddings – summarised representation for an entire sentence.
• Document embeddings – for long texts.

4.5 Transformer-Based Embeddings

Transformers use self-attention to create a context-aware representation for each token. This allows models to “attend” to important words in the sequence and incorporate global context.

4.6 Embedding Spaces & Geometry

Contextual embeddings exhibit rich geometric structure:

• Clusters reflect semantic categories.
• Vector directions encode relationships.
• Norms correlate with information content.

Page 5 – Practical Use of Embeddings, Limitations & Summary

5.1 Using Pretrained Embeddings

Common strategies:

• Load pretrained static embeddings (Word2Vec, GloVe).
• Use FastText for OOV (out-of-vocabulary) words.
• Extract contextual embeddings from BERT or similar models.

Pretrained embeddings dramatically improve performance in low-data settings.

5.2 Fine-Tuning & Feature-Based Approaches

Contextual encoders can be used:

• As fixed feature extractors – embedding + classical ML.
• Via fine-tuning – full model optimisation on task-specific data.

5.3 Efficiency Considerations

Static embeddings are computationally cheap. Contextual embeddings require substantial compute and memory.

5.4 Bias & Ethical Considerations

Embeddings inherit biases present in the data:

• Gender stereotypes
• Ethnic & cultural associations
• Socio-economic correlations

Mitigating techniques include:

• Debiasing in embedding space
• Dataset filtering
• Regular auditing

5.5 Summary

Module 3 introduced distributed representations and neural embeddings, tracing the evolution from classical count-based models to Word2Vec, GloVe, FastText, and contextual embeddings from Transformers.

These embeddings form the backbone of modern NLP systems and set the stage for Modules 4 and 5, which explore sequence models, attention, and Transformers in greater depth.