Text preprocessing and tokenization

A language model does not read sentences: it handles numbers. The first step of any NLP project is therefore to turn text into a sequence of token identifiers drawn from a fixed vocabulary.

Normalizing text

Normalization makes different spellings of the same word comparable:

import re
import unicodedata

def normaliser(texte):
    texte = texte.lower()                                   # case
    texte = unicodedata.normalize("NFD", texte)             # separates letters and accents
    texte = "".join(c for c in texte if unicodedata.category(c) != "Mn")
    texte = re.sub(r"\s+", " ", texte).strip()             # multiple spaces
    return texte

print(normaliser("  L'Été   est LÀ ! "))   # "l'ete est la !"

Every choice has a cost: removing accents confuses « pêche » (peach) and « pèche » (sins), lowercasing erases the difference between « Apple » and « apple ». For a TF-IDF classifier, strong normalization often helps; for a pre-trained Transformer model, you barely normalize: its tokenizer was designed for raw text. Essential rule: the same preprocessing must be applied in training and in production.

Tokenizing into words

The simplest approach splits on alphanumeric characters:

def tokeniser(texte):
    return re.findall(r"\w+", normaliser(texte))

print(tokeniser("Le colis n'est pas arrivé."))
# ['le', 'colis', 'n', 'est', 'pas', 'arrive']

Libraries such as spaCy handle elisions, numbers or dates finely, and add lemmatization (« arrivées » → « arriver »). The drawback of word tokens: the vocabulary explodes and any word absent from training (typo, proper noun, neologism) becomes unknown.

Subwords and BPE

Modern models split into subwords: frequent words stay whole, rare words are broken down (« anticonstitutionnellement » → « anti », « constitution », « nellement »). There is no unknown word anymore.

The BPE (Byte Pair Encoding) algorithm learns this vocabulary:

1. start with each word split into characters (or bytes, to cover all of Unicode);
2. count the adjacent pairs of symbols in the corpus, weighted by word frequency;
3. merge the most frequent pair into a new symbol;
4. repeat until the desired vocabulary size (often 32,000 to 200,000 tokens).

from collections import Counter

corpus = {("b", "a", "s"): 5, ("b", "a", "l"): 3, ("v", "a", "s"): 2}
paires = Counter()
for mot, freq in corpus.items():
    for a, b in zip(mot, mot[1:]):
        paires[(a, b)] += freq
print(paires.most_common(1))   # [(('a', 's'), 7)] -> we merge "a" + "s"

WordPiece (BERT) and Unigram (SentencePiece) follow the same idea with other merge criteria.

Model tokenizers

With Hugging Face, each model provides its tokenizer:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("camembert-base")
enc = tok("Le colis n'est pas arrivé.")
print(enc["input_ids"])                                   # identifiers
print(tok.convert_ids_to_tokens(enc["input_ids"]))        # word pieces

The number of tokens matters: it determines a model’s context limit and the cost of LLM APIs, which are billed per token. French generally produces slightly more tokens than English for the same content.

Key takeaways

Consistent normalization, tokenization suited to the model, and awareness of the number of tokens: these foundations condition everything else, from TF-IDF to LLMs.