Page 1 – Attention Mechanisms: Theory, Motivation, and Variants

1.1 Motivation for Attention

Recurrent and convolutional architectures face limitations in capturing long-range dependencies. Attention mechanisms address these limitations by allowing models to dynamically focus on relevant parts of the input.

1.2 General Attention Framework

Given a set of queries Q, keys K, and values V, attention computes:

Attention(Q, K, V) = softmax( QKᵀ / √dk ) V

Where:

• Query → the element seeking context.
• Key → points to relevant information.
• Value → the actual information extracted.

1.3 Scaled Dot-Product Attention

The scaling factor √dk ensures stable gradients by preventing large dot-product values.

1.4 Multi-Head Attention (MHA)

Instead of a single attention map, MHA learns multiple projections and attends jointly:

headᵢ = Attention(QWᵢ^Q, KWᵢ^K, VWᵢ^V)

MHA allows models to focus on different types of relationships simultaneously (e.g., syntax vs. semantics).

1.5 Types of Attention

• Self-attention: Q = K = V.
• Cross-attention: Q comes from decoder, K/V from encoder.
• Local attention for sequences with window constraints.
• Sparse attention for long sequences.

1.6 Benefits of Attention

• No need for recurrence → parallel computations.
• Captures global dependencies.
• Scales effectively with modern accelerators.

Page 2 – Transformer Architectures

2.1 Transformer Encoder–Decoder Structure

The transformer introduced by Vaswani et al. (2017) has two main components:

• Encoder: stacks of self-attention + feed-forward layers.
• Decoder: self-attention + cross-attention + feed-forward layers.

2.2 Positional Encoding

Since transformers lack recurrence, position information is injected via:

• Sine–cosine encodings.
• Learned positional embeddings.

2.3 Feed-Forward Networks (FFN)

Each transformer block contains an FFN applied independently to each position:

FFN(x) = ReLU(xW₁ + b₁) W₂ + b₂

2.4 Layer Normalisation

Used to stabilise training by normalising across features:

x̂ = (x − μ) / √(σ² + ε)

2.5 Residual Connections

Residual links allow deeper models and mitigate vanishing gradients:

Output = LayerNorm(x + SubLayer(x))

2.6 Transformer Variants

• BERT: bidirectional encoder for masked-language pretraining.
• GPT: autoregressive decoder-only transformer.
• T5: unified text-to-text framework.
• ViT: Vision Transformer for images.
• Swin Transformer: hierarchical, window-based attention.

2.7 Why Transformers Became Dominant

• Superior parallelism vs. RNNs.
• Scalable to billions of parameters.
• Effective transfer learning.
• Minimal inductive bias, allowing broad applicability.

Page 3 – Large Language Models (LLMs)

3.1 What Are LLMs?

Large Language Models (LLMs) are massive transformer-based architectures trained on diverse text corpora to learn statistical patterns in language.

3.2 Scaling Laws

Empirical research shows predictable improvements in performance as we scale:

• Model parameters.
• Dataset size.
• Compute budget.

LLMs such as GPT-4, PaLM, LLaMA, and Claude illustrate the power of scaling.

3.3 Pretraining Objectives

• Autoregressive (predict next token): GPT family.
• Masked language modelling (predict masked words): BERT, RoBERTa.
• Sequence-to-sequence prediction: T5.

3.4 Instruction Finetuning

LLMs are aligned to human instructions using:

• Supervised finetuning on instruction datasets.
• Reinforcement Learning from Human Feedback (RLHF).
• Preference modelling (DPO, PPO).

3.5 Emergent Capabilities

At scale, models exhibit:

• Few-shot learning.
• In-context learning.
• Chain-of-thought reasoning.
• Multimodal integration.

3.6 Limitations and Risks

• Hallucinations.
• Bias amplification.
• Lack of explainability.
• High compute and energy costs.

Page 4 – Training Large Models: Optimisation, Stability, and Scaling

4.1 Training Instabilities

Large models are sensitive to:

• Learning rate schedules.
• Weight initialisation.
• Activation distributions.
• Gradient norms.

4.2 Optimisation Techniques

• AdamW (decoupled weight decay).
• Layer-wise learning rate decay (LLRD).
• Warm-up and cosine decay schedules.
• Gradient clipping.

4.3 Regularisation for Large Models

• Dropout.
• Stochastic depth for deep stacks.
• Data augmentation (text and vision).

4.4 Data Parallelism

Many GPUs compute gradients independently, then synchronise updates.

4.5 Model Parallelism

For enormous models, parameters are split:

• Tensor parallelism: split internal matrices.
• Pipeline parallelism: split layers across machines.

4.6 Memory Optimisation Techniques

• Mixed precision training (FP16/BF16).
• Gradient checkpointing.
• Activation recomputation.

4.7 Retrieval-Augmented Models (RAG)

Large models can be augmented with retrieval mechanisms to access external databases, improving factual accuracy.

Page 5 – Applications, Future Frontiers, and Ethical Considerations

5.1 Applications Across Domains

• Healthcare: diagnosis, radiology, triage, drug discovery.
• Finance: risk modelling, fraud detection, forecasting.
• NLP: translation, summarisation, legal analysis.
• Vision: detection, segmentation, medical imaging.
• Robotics: policy learning, autonomous control.
• Education: tutoring systems, personalised learning.

5.2 Frontiers in Deep Learning Research

• Multimodal models (image + text + audio).
• Continual learning without catastrophic forgetting.
• Neural-symbolic systems integrating logic and learning.
• Energy-efficient architectures and green AI.
• Self-supervised learning for data-scarce domains.

5.3 Ethical and Societal Considerations

As deep learning systems permeate society, ethical challenges arise:

• Fairness and bias mitigation.
• Transparency and explainability.
• Privacy-preserving learning.
• Regulation and governance frameworks.

5.4 Responsible Deployment

Responsible AI requires:

• Auditability.
• Monitoring model drift.
• Human-in-the-loop oversight.
• Clear fail-safe mechanisms.

5.5 Summary of Module 5

This advanced module has explored:

• The theory and mechanics of attention.
• Transformer architectures and their variants.
• Large language models, scaling laws, and emergent abilities.
• Optimisation strategies for large-scale training.
• Modern applications and ethical considerations.

Collectively, Modules 1–5 establish a rigorous conceptual and practical foundation in machine learning, enabling further specialisation in areas such as deep generative modelling, reinforcement learning, or applied AI.