Understanding Tokenization Process in Simple Words

Artificial Intelligence, it is the manifestation of humans making something that will work for humans, like helping them in their day-to-day tasks, such as fixing their mistakes, writing emails, scheduling meetings, writing code, fixing grammatical errors, etc. Humans always push the boundaries to achieve something that will help them to ease their job or simplify their daily activities. If we look back at our history, we can find many examples that illustrate the same. AI is one such example. Nowadays, it feels like big companies are poised to create something that we have seen in our childhood movies, like Skynet destroying the world; Terminators are everywhere; humans are controlled by AI, like we have seen in movies like The Matrix. But don't worry; there is a difference between reality and fiction. Even if not everything from the movies will be true, some parts definitely will. But let's hope for a better future.

Yes, artificial intelligence technology is developed by studying the human mind. You might have come across terms like neural networks; they are developed based on how the human mind develops patterns when we learn something or recall something. Though we don't fully understand how the human mind works, still scientists studying the human brain. But here one question arises: how do these LLMs, or, say, neural networks, understand the world like we humans do? I mean, yes, to understand our surroundings, we have our five senses, or six, if you are a superhuman. But anyway, these senses help us to continuously monitor and feed data to our brain and evaluate the outcome. But in the case of AI, to understand any data, it does not follow the approach that our human brain does; instead, large language models follow a different approach in which tokenization plays an important role.

What is tokenization ? 

Tokenization is the process of breaking down text into smaller units called tokens, which are the basic building blocks an LLM actually processes. Neural networks work with numbers, not raw text, so tokenization is the bridge between human language and the numerical representations the model understands. These tokens captures a languages minimum semantic meaningful unit.

What is the idea behind the tokenization?

1. Splitting text: Input text is broken into tokens — these can be whole words, parts of words (subwords), individual characters, or even punctuation marks, depending on the tokenization method.
2.  Mapping to IDs: Each unique token is mapped to a numerical ID via a fixed vocabulary (e.g., "cat" → 1543).
3. Embedding: Those IDs are then converted into vectors (embeddings) that the model uses internally for computation.
4. Reverse process: When generating output, the model predicts token IDs, which are then converted back into readable text.

Though above is the idea behind the tokenization there are different types of tokenization followed depending upon what you want to achieve. 

1. Word-level: Each word is a token. Simple, but leads to huge vocabularies and struggles with rare/unseen words.
2. Character-level: Each character is a token. Small vocabulary, but sequences become very long and lose semantic efficiency.
3. Sub-word-level (most common in modern LLMs): Words are broken into meaningful chunks. For example, "unhappiness" might become "un" + "happi" + "ness". This balances vocabulary size with the ability to handle rare/unseen words. 

Popular algorithms used for tokenization:

• BPE (Byte Pair Encoding) - used by GPT models
• Word Piece - used by BERT
• Sentence Piece / Unigram - used by many multilingual models

Understanding tokenization in detail

Since now you know the definition of tokenization and what is the idea behind it lets discuss the tokenization in detail. As we already have some idea that tokens are nothing but small units of a bigger sentence which are tokenized based on algorithmic rules, but LLMs can't consume the splitted units of sentence directly since they work based on matrix based operations (in simple words they use math), so in order to make the tokens to be consumed by LLMs they first need to be converted into some numeric representations. Since we now know that tokens are not words but parts of words known as subwords. Which are kind of representation that lies somewhere in between words and letters. 

Tokenization is the feature engineering for the LLMs, its because tokens are the first thing that large language models deals with, and they need to be processed carefully so that the underlying algorithms can id them and can create the vocabulary (we will talk about this more while moving ahead don't worry !). The larger the vocabulary with unique tokens, better the LLM will understand semantic relationship with the words and will process the information successfully.

The below diagram shows the basic tokenization process in detail, have a look - 

As you can see we need input to process firs, then, we need a numeric identifier for each token to store and retrieve the information you will associate with that token. While the middle layers are something we can choose what can happen.

The last step of tokenization process is to build the vocabulary. Vocabulary of a model is the number of unique tokens seen during the training process when we feed data to an algorithm. It always takes a large amount of data to build a vocabulary enriched with unique tokens, which also determines how better a LLM can understand your input. But there are some trade-offs are also there while creating vocabulary for  your LLM. When you build a large vocabulary there are chances, because of its size it may consume more computational power or if not more memory or disk space, which make it harder to deploy or transfer it to other machines, vocabulary size is one factor contributing to an LLM’s size, so discussing methods and tradeoffs for controlling vocabulary size is vital.

Here is a clear, structured section you can insert into your blog post to explain how modern tokenizers work and why Byte Pair Encoding (BPE) is the go-to method for today’s Large Language Models:

How Modern Tokenizers Work & The Role of BPE

Modern Large Language Models (LLMs) rely on subword tokenization, a hybrid approach that strikes a balance between word-level and character-level tokenization.

• Word-level tokenization struggles with massive vocabularies and unknown words (out-of-vocabulary terms).
  
  
• Character-level tokenization keeps the vocabulary small, but produces extremely long sequences that waste compute and lose semantic context.
  
  

Sub-word tokenization solves both issues: common words remain intact (e.g., "token"), while rare or complex words are split into meaningful chunks (e.g., "tokenization" → "token" + "ization").

How Byte Pair Encoding (BPE) Works

Byte Pair Encoding (BPE) is one of the most widely used algorithms for training subword tokenizers (used by models like GPT-2, GPT-3, GPT-4, and LLaMA). Originally a data compression technique, BPE builds a vocabulary by iteratively merging the most frequent character pairs in a dataset.

Step-by-Step Breakdown:

1. Initialization:
   The dataset is split into individual characters, marking the end of words with a special character (e.g., </w>). Every unique character becomes the starting vocabulary.
   
   
2. Frequency Counting:
   The tokenizer scans the dataset to find the most frequently occurring adjacent pair of characters or sub-words.
   
   
3. Merging:
   The most frequent pair is merged into a single new token and added to the vocabulary.
   
   
4. Iteration:
   Steps 2 and 3 repeat thousands of times until the vocabulary reaches a pre-defined target size (e.g., 32,000 or 50,000 tokens).
   
   

Simple Example:

Suppose your training text contains: cost, costing, frequent, frequently.

• Step 1: Vocabulary starts with individual characters: c, o, s, t, i, n, g, f, r, e, q, u, l, y.
  
  
• Step 2: The pair s + t appears frequently across cost, costing, and frequent.
  
  
• Step 3: The tokenizer merges s and t into a single token: st.
  
  
• Step 4: Next, c + o and st might merge to form cost.
  
  

Eventually, common words like cost stay complete, while suffixes like ing or ly become reusable tokens.

Why BPE is Essential for Modern LLMs

• Zero Out-of-Vocabulary (OOV) Errors: If the tokenizer encounters a word it has never seen before, it simply breaks it down into familiar subwords or individual characters.
  
  
• Efficient Memory & Compute: BPE dramatically shortens the sequence length compared to character tokenization while keeping the vocabulary compact compared to word-level tokenization.
  
  
• Byte-Level BPE Flexibility: Modern variants (like Byte-Level BPE used in GPT-4) operate directly on raw bytes rather than Unicode characters. This allows the model to handle any language, code, formatting, or emoji without crashing or failing on unknown symbols.