Learning NLP From the Ground Up

Why Start From Scratch?

There is a temptation in NLP to start with HuggingFace, load a pretrained model, and call it done. That works. But it does not teach you anything about what is actually happening when text gets turned into numbers, why word vectors encode meaning the way they do, or what attention mechanisms are actually computing.

These two repos document a deliberate decision to go the other way: start from the very bottom and build upward, understanding each layer before moving to the next.

Stage 1: Tokenisation and Basic NLP Primitives

The starting point was the most fundamental question in NLP: how do you turn text into something a machine can work with?

The first notebook starts with spaCy for tokenisation, exploring the full pipeline: tok2vec, tagger, parser, attribute_ruler, lemmatizer, ner, and understanding what each component contributes. The dependency parser was visualised using displacy.serve, making the grammatical relationships in sentences like the opening lines of The Book Thief explicit and visible.

An important early lesson: you can disable pipeline components you do not need for speed. nlp.pipe_names shows what is active. If you only need tokenisation, disabling the parser and NER saves significant time on large corpora.

The next step was Bag of Words with CountVectorizer: converting documents to term frequency vectors. Simple, lossy, but a clean foundation for understanding what information is and is not preserved when text becomes a matrix.

Stage 2: TF-IDF and Spam Detection

The second notebook tackled a real classification task: SMS spam detection on a 4,837 message dataset.

Rather than using a trained classifier, this implementation uses TF-IDF vectors with a centroid-based scoring approach:

1. Fit a TfidfVectorizer with casual tokenisation (handling SMS-style abbreviations)
2. Compute a spam centroid: the mean TF-IDF vector across all spam messages
3. Compute a ham centroid: the mean TF-IDF vector across all legitimate messages
4. Score each document by its dot product with spam_centroid - ham_centroid

The result is a single spamminess score per message. Threshold at 0.5 and you have a spam classifier with no training loop, no gradient descent, just vector geometry.

Stage 3: Word Vectors and Semantic Reasoning

The third notebook explored word2vec using the Google News vectors (3,000,000 words, 300-dimensional embeddings) via the nlpia library, working through Chapter 6 of Natural Language Processing in Action.

The goal was to understand what word vectors actually encode. Not just "similar words are close together" but the geometry of meaning: how vector arithmetic captures semantic relationships, why king - man + woman ≈ queen, and what it means to reason with word vectors rather than with words.

The nessvector concept from the NLPIA book was explored here: decomposing a word vector into named semantic components to understand what dimensions of meaning the embedding has captured for a given entity.

Working with the Google News vectors at this scale (3M vocabulary, 300 dimensions) also gave a practical sense of the memory and compute requirements of pre-transformer NLP systems.

Stage 4: Building Embeddings From Scratch

Rather than just using pretrained embeddings, the next step was building an embedding layer from scratch: understanding exactly what the embedding matrix is doing and why.

The key insight that required working through: a sentence cannot simply be converted into a single 1D array directly. The correct approach is to convert each word in the sentence to a one-hot vector with respect to the full vocabulary, then stack those vectors to represent the sentence. Word order is preserved through the stacking order.

The EmbeddingFunctions notebook implements this from scratch using sparse matrices (scipy.sparse.csr_matrix) rather than dense arrays. A critical practical decision when your vocabulary is large: a dense one-hot vector for a 50,000-word vocabulary is a 50,000-dimensional vector of mostly zeros. A sparse representation stores only the non-zero index, making it memory-efficient enough to encode an entire corpus.

Key functions built:

• createWordOhe: creates a sparse one-hot vector for a single word given its vocabulary index
• createSentenceOhe: tokenises a sentence, looks up each word in the vocabulary dictionary, builds and stacks the one-hot vectors in sentence order
• delete_row_csr: removes a row from a CSR sparse matrix in-place (required to strip the empty initialisation row, a non-trivial operation that required a specific approach from the sparse matrix documentation)
• corpusEncoder: encodes an entire corpus to disk as .npz sparse matrix files, one per sentence

Working with BookCorpus at this scale (74,004,228 sentences, ~4.8GB) meant that encoding strategy was not academic. Saving each sentence as a compressed sparse .npz file was the practical solution to fitting the encoded corpus in a reasonable footprint.

Stage 5: Skip-Gram Word2Vec

With one-hot encoding working, the next step was training a word embedding from scratch using the skip-gram approach: given a word, predict its surrounding context words.

The EmbeddingsDataloaders notebook implements the skip-gram data pipeline with a configurable context window. For each word t in a sentence, targets are (t-2), (t-1), (t+1), (t+2). The context window size is double the window parameter, so window=2 gives 4 context predictions per input word.

A custom PyTorch Dataset class handles the data loading, drawing on the sparse encoded corpus files and the vocabulary dictionary built in the previous stage.

The embedding network (EmbeddingLayerNetwork) is a three-layer fully connected network with a bottleneck architecture:

Input (vocab_size) → Hidden → Embedding size → Output (vocab_size)

The embedding layer is the bottleneck. The network is trained to predict context words from input words, and in doing so, the weights of the bottleneck layer learn to encode semantic relationships. After training, those weights are the embedding matrix: the same principle behind Word2Vec.

Stage 6: Positional Encoding

Moving toward sequence models and eventually transformers, the next piece to understand was positional encoding: how do you give a model information about the position of each token in a sequence without disrupting the embedding values?

The PositionalEncoding module from PyTorch's transformer tutorial was implemented and studied in detail:

position = torch.arange(max_len).unsqueeze(1)
div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
pe[:, 0, 0::2] = torch.sin(position * div_term)
pe[:, 0, 1::2] = torch.cos(position * div_term)

The key insight: sine and cosine at different frequencies encode position as a unique pattern across the embedding dimensions. The model can learn to attend to position relationships because nearby positions have similar encodings and distant positions have distinguishably different ones. The register_buffer call stores the positional encoding matrix as a non-trainable parameter: it moves with the model to whatever device it is on, but is never updated by the optimiser.

Working through the shapes explicitly (pe_, pe_sin, pe_cos) at each step made the construction concrete rather than abstract.

Stage 7: LSTM Sequence Model

Before the full attention mechanism, a multi-layer LSTM (MV_LSTM_v2_dropout1) was built as the sequence processing backbone:

• Configurable number of layers and hidden size (default 512)
• Dropout between layers
• Hidden and cell states initialised to zero per batch
• Batch-first input format

This LSTM then became the temporal processing component in the attention network built in the next stage.

Stage 8: Global Attention Network

The final architecture assembled all the previous components into a BasicGlobalAttentionNetwork: a sequence model with a global attention mechanism.

The network takes a vocabulary-length input, passes it through an embedding layer (fc1), feeds the embedded sequence into the LSTM, then applies attention over the LSTM outputs using separate key, query, and value projection networks:

self.key_neurons   = key_neurons
self.query_neurons = query_neurons
self.value_neurons = value_neurons

This is the same key-query-value structure that underlies the transformer attention mechanism, implemented as a standalone module on top of an LSTM. The attention weights determine which parts of the sequence the model should focus on when producing the final output.

The Thread

Looking back across both repos, the progression is a complete bottom-up reconstruction of the ideas that underlie modern NLP:

Text → Tokens → Bag of Words → TF-IDF → One-Hot → Skip-Gram Embeddings
     → Positional Encoding → LSTM → Attention → Transformer (direction of travel)

None of it was taken on faith. Each piece was implemented, broken, debugged, and understood before moving to the next. That foundation is what makes working with systems like System Zero's hybrid retrieval pipeline, or reasoning about embedding model selection for different domains, feel grounded rather than abstract.

related project

System Zero: Building a Self-Hosted Agentic RAG Platform From Scratch

NLP Projects: github.com/Kikumu/NLP-projects

Journey with NLP: github.com/Kikumu/Journey-with-NLP