AI Tokens

Why Tokens?

Unlike the traditional way of storing and retrieving data, the AI Model shall build relationships between the text and generate text with the right context, so it requires a special data structure (Vectors) to store it in a unique way (multi-dimensional arrays). 

When we have text as input in many use cases, it has to be converted into tokens first and then into vectors. A token is a chunk of text the model processes. A token indeed can be a word, a subword, or a character. 

The Process

Step-1: Text → Tokens (Tokenization)

Before anything else, the model uses a tokenizer (often based on Byte Pair Encoding (BPE) or similar methods).

Example:

Why split like this?

• Reduces vocabulary size
• Helps the model understand new/rare words

Step-2: Tokens → Numbers (Token IDs)

Each token is mapped to a unique number:

Now the sentence becomes:

Step-3: Numbers → Vectors (Embeddings)

Each token ID is converted into a vector (a list of numbers):

These vectors capture meaning:

• “cat” and “dog” → close together
• “cat” and “car” → far apart

Step-4: Model predicts next token

The model:

1. Looks at previous tokens
2. Calculates probabilities for the next token

Example

Input: "The house is very"
Output probabilities:

• "big" → 40%
• "small" → 25%
• "beautiful" → 20%

Step-5: Token generation

The model chooses one token (based on probability + randomness settings):

Then repeats:

This loop continues → token by token generation

Entire flow

Check out relevant topics

The World of Vectors in AI

Why are Vectors a big deal for AI? Understand what we have been doing so far vs what we can get via AI Vectors as a perfect data type and format. What are we doing to retrieve the data? 1. RDBMS options 2. NoSQL and other database options 3. Search

expectations.aiRavi Kalidindi