Page 1 – Foundations of Neural Networks

1.1 Biological Inspiration

Artificial neural networks are inspired by the structure of biological neurons, which process inputs from upstream neurons and transmit electrical signals depending on activation thresholds. Although the analogy is loose, the conceptual mapping is:

• Inputs → dendrites
• Weighted sum → synaptic strength
• Activation function → neuron firing
• Output → axon signal

1.2 Perceptron Model

The simplest neural unit is the perceptron. Given an input vector x, weights w, and bias b, the perceptron computes:

z = wᵀx + b

followed by an activation function σ(z), producing the output:

ŷ = σ(z)

1.3 Activation Functions

Activation functions introduce non-linearity, enabling networks to learn complex mappings. Common choices:

• Sigmoid: σ(z) = 1 / (1 + e−z)
• Tanh: tanh(z)
• ReLU (Rectified Linear Unit): ReLU(z) = max(0, z)
• Leaky ReLU
• Softmax (for multi-class output)

1.4 Multi-Layer Perceptron (MLP)

A multi-layer perceptron (MLP) consists of multiple layers of neurons arranged sequentially:

• Input layer: raw features.
• Hidden layers: intermediate representations.
• Output layer: prediction.

The forward pass through an MLP with one hidden layer is:

h = σ₁(W₁x + b₁)
ŷ = σ₂(W₂h + b₂)

1.5 Universal Approximation Theorem

A neural network with a single hidden layer and a non-linear activation function can approximate any continuous function on a compact domain, given enough hidden units. Although the theorem does not guarantee efficient learning, it supports the expressive power of neural networks.

Page 2 – Training Neural Networks: Gradients and Backpropagation

2.1 Loss Functions

Neural networks are trained by minimising a loss function:

• MSE for regression.
• Binary cross-entropy for binary classification.
• Categorical cross-entropy for multi-class tasks.

2.2 Gradient-Based Learning

Networks are parameterised by weights and biases. We update parameters using gradient descent:

θ ← θ − η ∇θ L(θ)

2.3 Backpropagation Overview

Computing gradients directly using the chain rule is tedious for deep networks. Backpropagation automates this process:

1. Forward pass: compute predictions and store intermediate values.
2. Backward pass: propagate gradients from output to input.
3. Update parameters using gradients.

2.4 Chain Rule in Vector Form

Given a network layer:

a = Wz + b  
h = σ(a)

gradients are computed as:

∂L/∂W = (∂L/∂h) ∘ σ'(a) zᵀ
∂L/∂b = (∂L/∂h) ∘ σ'(a)

2.5 Vanishing and Exploding Gradients

Deep networks can suffer from:

• Vanishing gradients: gradients shrink during backprop.
• Exploding gradients: gradients become excessively large.

Solutions:

• ReLU-family activations.
• Batch normalisation.
• Residual connections.
• Weight initialisation strategies (Xavier, He-initialisation).

2.6 Optimisers

Beyond basic gradient descent, common optimisers include:

• SGD with momentum.
• Adam (adaptive moments).
• RMSProp.
• AdaGrad.

Page 3 – Deep Network Architectures

3.1 Deep vs. Shallow Networks

A network is considered “deep” if it contains multiple hidden layers. Depth allows hierarchical feature extraction:

• Lower layers learn local or simple patterns.
• Higher layers learn abstract, composite, or semantic patterns.

3.2 Convolutional Neural Networks (CNNs)

CNNs are designed for image and spatial data. They use convolutional filters to extract local features:

hi,j = Σ W ∘ region(x)

Features:

• Weight sharing → fewer parameters.
• Translation invariance.
• Hierarchical filters (edges → textures → objects).

3.3 Recurrent Neural Networks (RNNs)

Designed for sequence data (text, time series). Maintain a hidden state:

hₜ = σ(Wxₜ + Uhₜ₋₁ + b)

Variants:

• LSTM (Long Short-Term Memory).
• GRU (Gated Recurrent Unit).

3.4 Transformers

Transformers use self-attention rather than recurrence. They dominate NLP and are expanding to vision and multimodal learning.

Attention(Q, K, V) = softmax(QKᵀ / √d) V

3.5 Autoencoders

Autoencoders learn compressed representations:

Encoder:   z = f(x)
Decoder:   x̂ = g(z)

Uses:

• Dimensionality reduction.
• Anomaly detection.
• Denoising.

3.6 Generative Models

Models that can generate new samples:

• GANs (Generative Adversarial Networks).
• VAEs (Variational Autoencoders).
• Diffusion models (modern state-of-the-art).

Page 4 – Regularisation, Generalisation, and Training Stability

4.1 Overfitting in Deep Networks

Deep networks can overfit due to large capacity. Common symptoms include:

• Training loss decreasing while validation loss increases.
• High variance in predictions.

4.2 Dropout

Dropout randomly zeroes out hidden units during training:

hᵢ ← 0 with probability p

This prevents co-adaptation and improves generalisation.

4.3 Batch Normalisation

Normalises activations across the batch:

â = (a − μ) / √(σ² + ε)

Benefits:

• Stabilises training.
• Allows higher learning rates.
• Acts as a regulariser.

4.4 Early Stopping

Stop training when validation loss stops improving for specified epochs.

4.5 Weight Regularisation

Similar to earlier modules:

• L2 regularisation.
• Weight decay (commonly used with AdamW).

4.6 Data Augmentation

Especially useful in vision and audio tasks:

• Random cropping.
• Rotation, flipping.
• Noise injection.
• Mixup and CutMix.

4.7 Generalisation in Deep Learning

Deep networks generalise well even in over-parameterised regimes. Theoretical understanding involves:

• Implicit bias of SGD.
• Flat vs. sharp minima.
• Double descent curves.

Page 5 – Implementation Workflow, Pipelines, and Practical Deep Learning

5.1 Data Pipelines

Effective deep learning requires efficient data ingestion:

• Batching data.
• Shuffling to avoid bias.
• Parallel data loaders.
• Prefetching.

Frameworks: PyTorch DataLoader, TensorFlow tf.data.

5.2 GPU Acceleration

Neural networks rely heavily on parallel computation. GPUs (and TPUs) accelerate:

• Matrix multiplications.
• Convolutions.
• Backpropagation.

5.3 Model Checkpointing

Save model weights during training to:

• Avoid losing progress from interruptions.
• Enable early stopping and rollback.

5.4 Monitoring and Visualisation

Tools such as TensorBoard or Weights & Biases help track:

• Loss curves.
• Learning rates.
• Activation distributions.
• Model graphs.

5.5 Deployment Considerations

• Export to ONNX or TorchScript.
• Optimise using TensorRT.
• Quantisation for mobile deployment.
• Pruning for efficiency.

5.6 Summary

In this module, we introduced:

• The structure and intuition behind neural networks.
• Activation functions and multi-layer perceptrons.
• Backpropagation and gradient optimisation.
• Deep architectures (CNNs, RNNs, Transformers).
• Regularisation and generalisation techniques.
• End-to-end deep learning workflow and deployment insights.

Module 5 will extend into Advanced Topics in Deep Learning, such as attention mechanisms, large language models, training stability, and modern architectures dominating current AI applications.