Page 1 – Text Preprocessing Fundamentals

1.1 Why Preprocessing Matters

Before text can be used by learning algorithms, it must be cleaned, normalised, and structured. Preprocessing reduces noise, enforces consistency, and prepares text for tokenisation, parsing, vectorisation, and modelling.

1.2 Common Preprocessing Steps

Typical preprocessing includes:

• Case normalisation – converting text to lowercase.
• Removing noise – punctuation, URLs, HTML tags.
• Unicode normalisation – handling accented characters.
• Expanding contractions – e.g., “don’t” → “do not”.
• Standardising numbers – replacing digits with placeholders.

1.3 Tokenisation

Tokenisation divides text into meaningful units. NLP models assume consistent token boundaries.

• Whitespace tokenisation: simple but fails on Chinese, Thai, and punctuation.
• Rule-based tokenisation: handles punctuation, hyphens, contractions.
• Statistical tokenisers: trained on annotated corpora.
• Subword tokenisation (BPE, WordPiece, SentencePiece): used in Transformers and LLMs.

1.4 Sentence Segmentation

Identifying sentence boundaries is crucial for translation, summarisation, and syntactic parsing. Naïve strategies (splitting on periods) often fail for abbreviations or titles.

1.5 Stopword Handling

Stopwords (e.g., “the”, “is”, “and”) may be removed in classical NLP pipelines to reduce noise, but modern deep learning models often retain them because they help capture syntactic structure.

1.6 Stemming vs. Lemmatization

• Stemming – rule-based truncation to remove affixes (e.g., “running” → “run”).
• Lemmatization – uses vocabulary and morphology to produce a valid root word.

Lemmatization is more accurate but slower; stemming is faster but crude.

Page 2 – Feature Engineering for Classical NLP

2.1 Why Representations Matter

Text must be converted to numerical features for machine learning. Good representations preserve linguistic meaning and structure while reducing sparsity and dimensionality.

2.2 Bag-of-Words (BoW)

BoW represents documents as unordered collections of words. It is simple, efficient, and performs well for classification tasks with linear models.

• Dimensionality = vocabulary size
• No word order or context
• Often combined with TF–IDF weighting

2.3 N-grams

N-grams capture limited word order by grouping sequences of n tokens.

• Unigrams – single words
• Bigrams – two-word sequences
• Trigrams – three-word sequences

N-grams improve modelling of local syntax but explode in dimensionality.

2.4 Term Frequency–Inverse Document Frequency (TF–IDF)

TF–IDF weights words by their importance:

• TF: how frequently a word appears in a document.
• IDF: how rare it is across documents.

This reduces weight for common words and emphasises topic-relevant ones.

2.5 Normalisation and Smoothing

Classical statistical models (e.g., n-gram LMs) require smoothing to prevent zero probabilities:

• Additive smoothing
• Kneser–Ney smoothing
• Good–Turing estimates

2.6 Feature Selection

Large vocabularies create sparse vectors. Feature selection reduces noise.

• Mutual Information
• Chi-Square test
• Document frequency thresholds
• Information gain

Page 3 – Classical NLP Models

3.1 Logistic Regression for NLP

Logistic regression is widely used for text classification (sentiment analysis, spam detection, topic classification).

Advantages:

• Strong baseline
• Handles high-dimensional sparse vectors
• Interpretable weights

3.2 Naïve Bayes Classifiers

Assumes conditional independence of features. Surprisingly effective for text.

• Multinomial Naïve Bayes – most common for NLP.
• Bernoulli Naïve Bayes – binary word presence.

3.3 Support Vector Machines (SVMs)

SVMs with linear kernels perform extremely well on TF–IDF and n-gram features.

• Effective in high-dimensional spaces
• Robust to overfitting
• Standard for many classification competitions pre-neural era

3.4 Hidden Markov Models (HMMs)

HMMs are probabilistic sequence models used for:

• POS tagging
• NER
• Speech recognition

They model:

• Transition probabilities between latent states
• Emission probabilities for observed words

3.5 Conditional Random Fields (CRFs)

CRFs are discriminative sequence models used extensively for NER and POS tagging before deep learning.

Unlike HMMs, CRFs:

• Model conditional probability directly
• Allow overlapping and arbitrary feature functions
• Outperform generative models when features are informative

3.6 Limitations of Classical Models

Classical models rely heavily on:

• Manual feature engineering
• Sparse high-dimensional representations
• Local context (n-grams)

They struggle with:

• Long-range dependencies
• Polysemy and semantic ambiguity
• Context-sensitive meaning

Page 4 – Statistical Language Models

4.1 What Is a Language Model?

A language model estimates the probability of a sequence of words: P(w₁, w₂, …, wₙ).

In classical NLP, language models were built using n-grams.

4.2 N-gram Language Models

The Markov assumption reduces complexity:

P(wₙ | w₁:ₙ₋₁) ≈ P(wₙ | wₙ₋₁, …, wₙ₋ₖ)

4.3 Smoothing Methods

Smoothing addresses the problem of unseen n-grams:

• Add-k smoothing – simplest
• Kneser–Ney smoothing – state-of-the-art for classical LMs
• Good–Turing – adjusts counts for rare terms

4.4 Perplexity

Perplexity measures how well a language model predicts test data:

PP = exp(-1/N Σ log P(wᵢ))

4.5 Limitations of N-gram Models

• Cannot model long-range context
• Exploding vocabulary size
• Sparse probability estimates
• No semantic representation

4.6 Transition to Neural Language Models

These limitations motivated the shift toward:

• Neural embeddings
• Recurrent networks (RNNs)
• Attention mechanisms
• Transformers

Page 5 – Practical NLP Pipelines & Implementation Considerations

5.1 A Typical Classical NLP Pipeline

1. Text cleaning and normalisation
2. Tokenisation and sentence segmentation
3. Feature extraction (BoW, TF–IDF, n-grams)
4. Model training (logistic regression, SVM, Naïve Bayes)
5. Evaluation (accuracy, F1-score)
6. Error analysis and iteration

5.2 Handling Noisy Text

Real-world text (social media, user reviews) contains:

• Misspellings
• Emojis
• Mixed languages
• Slang and abbreviations

5.3 Domain Adaptation

Models trained on formal newswire may fail on conversational text. Strategies include:

• Corpus balancing
• Re-training with in-domain data
• Adaptation through fine-tuning (in neural era)

5.4 Feature Drift

Vocabulary and semantics evolve over time. Classical models may degrade without periodic retraining.

5.5 Interpretability

Classical models are highly interpretable compared to modern neural models. Weights in logistic regression or SVMs reveal:

• Key discriminative words
• Biases in training data
• Topic clusters

5.6 Summary

Module 2 has covered the essential transformations that convert raw text into usable numerical features for machine learning models. These techniques underpin classical NLP systems and provide the conceptual foundation for understanding neural representations and embeddings in Module 3.