Page 1 – Sequence Modelling & Recurrent Neural Networks

1.1 Why Sequence Models?

Many NLP tasks involve sequential data where order matters:

• Language modelling
• Machine translation
• Speech recognition
• Named entity recognition (NER)
• Dialogue modelling

Sequence models must maintain information across time steps and produce context-aware representations.

1.2 Recurrent Neural Networks (RNNs)

RNNs process sequences by maintaining a hidden state:

hₜ = f(Wxh xₜ + Whh hₜ₋₁ + b)

This recurrent connection enables:

• Information flow through time
• Capturing short-term dependencies
• Sequential prediction

1.3 Forward Propagation Through Time

At each time step, the RNN updates:

1. Hidden state (memory)
2. Output (prediction)

1.4 Backpropagation Through Time (BPTT)

RNN training uses BPTT, where gradients are propagated through all time steps, allowing earlier states to influence later predictions.

1.5 Vanishing & Exploding Gradients

Classic RNNs suffer from:

• Vanishing gradients – long-term dependencies fade.
• Exploding gradients – unstable training.

These problems motivated gated architectures (LSTMs & GRUs).

1.6 Bidirectional RNNs

Bidirectional RNNs process sequences in both directions:

• Forward RNN: left → right
• Backward RNN: right → left

Useful for POS tagging, NER, sentence classification, etc.

Page 2 – LSTM and GRU Architectures

2.1 Long Short-Term Memory (LSTM)

LSTMs address vanishing gradients using gating mechanisms that regulate information flow.

2.2 LSTM Gates

• Forget gate – what to erase
• Input gate – what to store
• Output gate – what to emit

LSTM equations:

fₜ = σ(Wf [hₜ₋₁, xₜ] + bf)
iₜ = σ(Wi [hₜ₋₁, xₜ] + bi)
oₜ = σ(Wo [hₜ₋₁, xₜ] + bo)
c̃ₜ = tanh(Wc [hₜ₋₁, xₜ] + bc)
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ
hₜ = oₜ ⊙ tanh(cₜ)
    

2.3 Advantages of LSTMs

• Learn long-range dependencies.
• Stable training vs. vanilla RNNs.
• Used in early seq2seq models, speech recognition, NLP pipelines.

2.4 Gated Recurrent Units (GRUs)

GRUs simplify LSTMs by combining gates into:

• Update gate
• Reset gate

2.5 GRU vs. LSTM

• GRUs train faster.
• Often perform comparably.
• Preferable for limited data.

2.6 Limitations of Gated RNNs

• Sequential computation → slow training.
• Poor scalability for very long sequences.
• Difficulty modelling long-range global dependencies.

These issues led to the development of attention and Transformers.

Page 3 – Attention Mechanisms

3.1 Why Attention?

RNNs struggle with long-range dependencies. Attention enables models to focus selectively on relevant parts of the input sequence.

3.2 Key–Value–Query (Q, K, V)

Attention uses three components:

• Query (Q) – what we want to focus on.
• Key (K) – labels for each position.
• Value (V) – encoded information.

3.3 Scaled Dot-Product Attention

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V
    

Dividing by √dₖ stabilises gradients.

3.4 Self-Attention

Self-attention computes attention within the same sequence:

• Each token attends to all other tokens.
• Captures long-range context efficiently.
• Forms the backbone of Transformer models.

3.5 Multi-Head Attention

Uses several attention heads in parallel, allowing the model to learn multiple types of relationships.

3.6 Benefits of Attention

• Parallel computation.
• Global receptive field.
• Better long-range modelling than RNNs.
• Interpretability through attention weights.

Page 4 – Encoder–Decoder Architectures

4.1 Motivation

Many NLP tasks require mapping an input sequence to an output sequence:

• Machine translation
• Summarisation
• Dialogue generation

4.2 Encoder

The encoder reads the input sequence and produces a sequence of hidden states.

4.3 Decoder

The decoder generates the output sequence step-by-step, conditioning on previous outputs.

4.4 Seq2Seq with RNNs

Early encoder–decoder models used LSTMs:

• Encoder compresses input into a fixed vector.
• Decoder expands vector into sequence.

Bottleneck: one vector cannot encode long sequences effectively.

4.5 Attention in Seq2Seq

Bahdanau & Luong attention improved seq2seq by allowing decoders to attend to all encoder states.

4.6 Positional Encoding

Transformers discard recurrence entirely; positional encodings inject sequence order using:

• Sinusoidal functions
• Learned embeddings

4.7 Encoder–Decoder with Transformers

Transformers replaced RNN seq2seq architectures thanks to:

• Self-attention
• Parallelism
• Long-range coherence

Page 5 – Applications, Limitations & Summary

5.1 Applications of Sequence & Attention Models

• Machine translation (Google Translate)
• Automatic speech recognition (ASR)
• Dialogue and chatbots
• Summarisation
• POS tagging, NER, dependency parsing
• Text classification

5.2 Limitations of RNN-Based Models

• Slow sequential computation
• Poor scalability
• Difficulty modelling long-range dependencies
• Bottlenecks in seq2seq without attention

5.3 Advantages of Attention & Transformers

• No recurrence → full parallelism
• Better learning dynamics
• Higher model capacity
• Better global context awareness
• State-of-the-art performance across NLP benchmarks

5.4 Training Considerations

• Large memory requirements
• Need for GPU/TPU acceleration
• Data hunger (especially for Transformers)

5.5 Ethical & Societal Considerations

Sequence-to-sequence and attention models power large-scale systems such as machine translation and conversational agents.

• Language bias and dialect under-representation
• Inappropriate generation / hallucination
• Propagating harmful stereotypes
• Use in surveillance or manipulation

5.6 Summary

Module 4 has covered the major architectures for sequence modelling— RNNs, LSTMs, GRUs, attention, and encoder–decoder systems. These models provide the conceptual foundation for understanding Transformers and Large Language Models (LLMs), which will be explored in Module 5.