Page 1 – What Is Natural Language Processing?

1.1 Motivation and Scope

Natural Language Processing (NLP) is a subfield of artificial intelligence that focuses on enabling computers to process, understand, generate, and interact using human language. It sits at the intersection of computer science, linguistics, and statistics, and underpins applications such as:

• Machine translation (e.g., translating between English and Chinese).
• Information retrieval and search engines.
• Question answering and virtual assistants.
• Text classification (spam detection, sentiment analysis).
• Summarisation and content generation.
• Dialogue systems and chatbots.

Unlike structured numerical data, natural language is ambiguous, context-dependent, and highly variable, making NLP a technically rich and conceptually nuanced domain.

1.2 From Rules to Data-Driven Methods

Historically, NLP systems were built using hand-crafted rules devised by linguists and domain experts. Early systems relied on:

• Manually curated grammars and lexicons.
• Pattern-matching and regular expressions.
• Symbolic reasoning and logic-based parsing.

While rule-based systems can be precise in narrow domains, they are brittle and difficult to maintain at scale. Modern NLP has shifted towards statistical and neural methods that learn patterns from large corpora, relying on:

• Probabilistic models (e.g., n-gram language models).
• Machine learning classifiers (e.g., logistic regression, SVMs).
• Neural architectures (RNNs, CNNs, Transformers, LLMs).

1.3 Language as Data

To process language computationally, we must represent it as data:

• Characters (Unicode code points).
• Tokens (words, subwords, morphemes).
• Sequences (sentences, paragraphs, documents).
• Structured annotations (POS tags, parse trees, entities).

The first step in most NLP pipelines is to map raw text into discrete or continuous representations that models can operate on. Subsequent modules will elaborate on feature representations and embeddings; here we focus on the conceptual space.

1.4 Core Challenges in NLP

NLP is difficult not merely because language is complex, but because it reflects the full variability of human cognition and culture. Key challenges include:

• Ambiguity: Words and sentences often admit multiple interpretations.
• Context dependence: Meaning depends on discourse and world knowledge.
• Variability: Dialects, registers, slang, and noisy input.
• Resource imbalance: Some languages or domains have limited labelled data.
• Pragmatics: Intent, politeness, and implicature are often implicit.

1.5 Example NLP Tasks

To make the discussion concrete, consider the following illustrative tasks:

• Sentiment analysis: Classify a review as positive, negative, or neutral.
• Named Entity Recognition (NER): Identify mentions of people, organisations, locations, and other entities in text.
• Machine translation: Translate “I am hungry” into Thai, or Chinese into English.
• Question answering: Given a passage and a question, extract or generate an answer.
• Dialogue management: Maintain coherent conversation over multiple turns.

These tasks will recur throughout the course as running examples to contextualise algorithms and representations.

Page 2 – Linguistic Concepts Relevant to NLP

2.1 Levels of Linguistic Analysis

Modern NLP systems implicitly or explicitly engage with several levels of linguistic structure:

• Phonology – sound patterns (less direct in text-based NLP).
• Morphology – internal structure of words.
• Syntax – structure of phrases and sentences.
• Semantics – meaning of words and compositions.
• Pragmatics – meaning in context, discourse, and interaction.

While neural models often learn representations without explicit linguistic labels, these levels remain conceptually useful for analysis and error diagnosis.

2.2 Morphology and Tokenisation

Morphology studies how words are formed from smaller units called morphemes. For example, “unhappiness” contains:

• un- (prefix, negation),
• happy (root),
• -ness (suffix, noun-forming).

Tokenisation strategies must account for morphology, particularly in:

• Highly inflected languages.
• Compounding (e.g., German, agglutinative languages).
• Scripts without whitespace boundaries (e.g., Chinese, Thai).

Subword tokenisation methods (Byte-Pair Encoding, WordPiece, SentencePiece) used in modern LLMs can be seen as data-driven approximations to morphological segmentation.

2.3 Syntax and Grammatical Structure

Syntax concerns how words combine into phrases and sentences. Syntax can be represented via:

• Constituency trees – nested phrase structure.
• Dependency trees – directed relations between heads and dependents.

Syntactic analysis is important for:

• Disambiguating sentence structure.
• Information extraction (subject–verb–object relations).
• Downstream tasks like QA, summarisation, and translation.

2.4 Semantics and Compositionality

Semantics is concerned with meaning. Classical semantics uses formal logic (e.g., predicate calculus) to model truth conditions. Compositionality is the principle that the meaning of a phrase is derived from the meaning of its parts and the rules used to combine them.

In practice, NLP often relies on:

• Distributional semantics – “You shall know a word by the company it keeps.”
• Vector embeddings – representing words and sentences as points in a high-dimensional space.
• Neural encoders – context-sensitive representations via RNNs or Transformers.

2.5 Pragmatics and Discourse

Pragmatics goes beyond literal word meaning to account for speaker intentions, politeness, implicature, and shared context. For example, “It’s cold in here” may function as a request to close the window.

Discourse spans multiple sentences or turns:

• Coreference (linking pronouns to entities).
• Coherence and discourse relations (contrast, cause, elaboration).
• Dialogue state and conversational context.

Large language models internalise many pragmatic and discourse regularities, but explicit modelling remains an active research area.

2.6 Linguistics and Data-Driven Models

While many modern NLP systems are trained end-to-end on annotated or unlabelled corpora, linguistic theory provides:

• A conceptual framework to understand what models might be learning.
• Diagnostic tools to probe and evaluate model behaviour.
• Guidance for designing inductive biases and structured architectures.

Page 3 – Text as Data: Corpora, Tokens, and Statistical Properties

3.1 Corpora in NLP

A corpus (plural: corpora) is a structured collection of texts used for training, evaluation, or analysis. Corpora can be:

• General-domain – newswire, web text, Wikipedia.
• Domain-specific – biomedical articles, legal documents, social media.
• Task-specific – parallel corpora for translation, labelled corpora for NER.

Important corpus attributes include:

• Size (tokens, documents).
• Language(s) and dialects.
• Temporal coverage.
• Annotation types and quality.

3.2 Tokens, Types, and Vocabulary

In text processing, it is useful to distinguish:

• Token – a particular occurrence of a word or symbol in the corpus.
• Type – a unique word form in the vocabulary.
• Vocabulary – the set of all observed types.

For example, in the sentence “The cat and the dog”, there are 5 tokens and 4 types (“the”, “cat”, “and”, “dog”) if we treat “The” and “the” as the same type (case-insensitive).

3.3 Zipf’s Law

Natural language corpora exhibit a characteristic frequency distribution known as Zipf’s law: the frequency of a word is approximately inversely proportional to its rank in the frequency list.

Consequences:

• A few words are extremely common (function words, stopwords).
• Many words are rare or occur only once (hapax legomena).
• Vocabulary growth continues as we add more data.

This heavy-tailed behaviour motivates:

• Subword representations to handle rare words and morphology.
• Smoothing techniques in language modelling.
• Careful evaluation on rare but important terms (e.g., entities).

3.4 Basic Preprocessing Operations

Before feeding text into models, we often apply:

• Normalisation – case-folding, Unicode normalisation.
• Tokenisation – splitting text into words or subwords.
• Sentence segmentation – identifying sentence boundaries.
• Stopword handling – optionally removing very common function words.
• Stemming / Lemmatization – mapping inflected forms to base forms.

Neural models may require less aggressive preprocessing, but understanding these steps remains important for diagnostics and smaller-scale systems.

3.5 Sparse vs. Dense Representations

Early NLP models used sparse representations such as:

• One-hot vectors (dimension = vocabulary size).
• Bag-of-words (term frequency, TF–IDF).

These representations ignore word order and can be extremely high-dimensional. Modern NLP uses dense embeddings that map tokens or sequences to lower-dimensional continuous vectors (e.g., 300–1024 dimensions), capturing distributional semantics and contextual information.

3.6 Annotation and Labelled Data

Many NLP tasks require labelled data:

• POS tags for part-of-speech tagging.
• Entity labels for NER.
• Sentiment labels for classification.
• Aligned sentence pairs for translation.

Annotation quality, inter-annotator agreement, and guidelines critically influence model performance and reproducibility.

Page 4 – Core NLP Tasks and Evaluation Metrics

4.1 Taxonomy of NLP Tasks

NLP tasks can be categorised along several dimensions:

• Level: word-level, sentence-level, document-level, discourse-level.
• Output type: classification, sequence labelling, sequence-to-sequence, ranking.
• Supervision: supervised, unsupervised, semi-supervised, self-supervised.

4.2 Word-Level and Sequence Labelling Tasks

• Part-of-Speech (POS) Tagging – assign syntactic category (noun, verb, etc.) to each token.
• Named Entity Recognition (NER) – label spans as PERSON, ORG, LOC, etc.
• Chunking – identify phrase segments (e.g., noun phrases).

These tasks typically produce a label sequence aligned with the token sequence.

4.3 Sentence-Level Tasks

• Text classification – sentiment analysis, topic classification.
• Natural Language Inference (NLI) – determine entailment, contradiction, or neutrality between sentence pairs.
• Paraphrase detection – identify if two sentences have similar meaning.

4.4 Sequence-to-Sequence Tasks

• Machine translation – map source sentence to target sentence.
• Summarisation – generate condensed versions of documents.
• Dialogue response generation – produce the next utterance.

4.5 Evaluation Metrics for NLP

Evaluation must align with task structure:

• Classification: accuracy, precision, recall, F1-score, macro/micro averaging for multi-class or imbalanced data.
• Sequence labelling: token-level or span-level F1 (e.g., CoNLL NER evaluation).
• Translation: BLEU, METEOR, chrF, COMET (combining learned metrics).
• Summarisation: ROUGE (n-gram overlap), BERTScore, and human evaluations (fluency, coherence, faithfulness).

4.6 Error Analysis

Quantitative metrics are necessary but not sufficient. Detailed error analysis helps identify:

• Systematic biases (e.g., failures on named entities or specific languages).
• Failure modes (e.g., negation, numerical reasoning, long-range dependencies).
• Domain mismatch between training and test data.

Structured error analysis is essential for academic work and robust deployment in high-stakes domains.

Page 5 – NLP Pipelines, Limitations, and Ethical Considerations

5.1 Classical NLP Pipeline

Traditional NLP systems are often implemented as pipelines:

1. Input normalisation – lowercasing, cleaning, script handling.
2. Tokenisation and sentence segmentation.
3. POS tagging and morphological analysis.
4. Parsing – syntactic and/or dependency parsing.
5. Semantic or task-specific layer – NER, sentiment, extraction, etc.

Errors can propagate along the pipeline; a mistake in tokenisation or tagging may impair downstream components.

5.2 End-to-End Neural Approaches

Modern neural NLP often collapses the pipeline into a single model that maps raw or lightly processed text to outputs, implicitly learning intermediate representations:

• Contextual embeddings via Transformers.
• Joint modelling of multiple tasks (multi-task learning).
• Pretraining + finetuning workflows.

This reduces manual feature engineering but requires large-scale data and compute, and can obscure internal reasoning.

5.3 Limitations of Current NLP Systems

Despite impressive performance, contemporary NLP systems have notable limitations:

• Brittleness to distribution shifts and adversarial inputs.
• Surface-level heuristics instead of deep understanding.
• Poor handling of world knowledge without augmentation.
• Difficulty with compositional generalisation and reasoning.
• Hallucination – generating fluent but incorrect content.

5.4 Bias, Fairness, and Societal Impact

NLP systems trained on large corpora inevitably absorb patterns present in the data, including:

• Gender, racial, and cultural stereotypes.
• Socio-economic and geopolitical biases.
• Imbalanced representation of languages or dialects.

Deployment in areas such as hiring, credit scoring, policing, or healthcare requires careful attention to:

• Dataset curation and documentation.
• Fairness metrics and auditing.
• Human oversight and contestability of automated decisions.
• Regulatory and ethical guidelines.

5.5 Responsible NLP Practice

Responsible NLP involves:

• Transparent reporting of datasets, models, and evaluation protocols.
• Awareness of limitations and clearly communicated caveats.
• Continuous monitoring and feedback loops post-deployment.
• Engagement with interdisciplinary perspectives (ethics, law, social sciences).

5.6 Summary and Forward Look

This introductory module has:

• Defined NLP and its core challenges.
• Introduced linguistic levels relevant to computational models.
• Discussed corpora, tokenisation, and statistical properties of text.
• Outlined major NLP tasks and evaluation metrics.
• Highlighted limitations and ethical considerations in modern NLP.

Subsequent modules will move from conceptual foundations to concrete modelling techniques, starting with text preprocessing, classical features, and baseline machine learning models for NLP tasks.