Przejdź do treści
promptowy.com / AI Glossary / Tokenization
Technology and architecture

Tokenization

Tokenization is the process of breaking raw text into smaller computational units, called tokens, before an AI model processes or generates language.

Tokenization is the foundational process in natural language processing where raw text is sliced into smaller fragments called tokens. Artificial intelligence models do not read sentences the way a student in London or a copywriter in New York reads them. Instead of understanding letters and grammar intuitively, an AI system relies purely on mathematical calculations, and tokenization serves as the bridge that turns written human language into numerical inputs.

Before a model learns to answer questions or write essays, it is trained alongside a tokenizer that builds a fixed dictionary of vocabulary units. When you submit text to the system, the tokenizer evaluates every character, space, and punctuation mark, matching pieces of your input to the numerical IDs stored in its pre-established vocabulary.

How tokenization works behind the scenes

Modern language models rarely split text strictly word by word or character by character. Splitting solely by full words would make the model vocabulary astronomically large, while splitting entirely by single letters would strip away context and demand too much computing power. Instead, systems use subword tokenization.

Under this approach, frequent English words like market, school, or report earn their own dedicated token. Rare words, specialized medical jargon, or misspelled phrases are disassembled into smaller chunks, syllables, or individual characters. For example, a straightforward word like warm might represent a single token, whereas an unusual variant like warmishness might be broken into two or three distinct subwords.

A practical example: From high street bakeries to offices

To see how this works in a business context, imagine a neighborhood bakery in Leeds or an independent logistics firm in Atlanta using automated customer service prompts. Consider how two similar inputs are processed:

  • Input A: "Please confirm delivery of fresh bread rolls."
  • Input B: "Plz confrm delvry of frsh brd rlls!"

In Input A, the words are standard and cleanly spelled. The tokenizer identifies common English terms, converting nearly every word into a single token. The prompt is compact, predictable, and straightforward for the system to process.

In Input B, the informal shorthand forces the tokenizer to work much harder. Because non-standard spellings like confrm or delvry do not exist as whole words in the tokenizer dictionary, the system fragments each word into multiple subword pieces or single letters. Even though Input B contains fewer total characters on screen, it actually generates a higher token count than Input A.

Why tokenization explains common AI quirks

Understanding tokenization clarifies several odd behaviors that users encounter when interacting with language models:

  • Letter counting errors: Users often notice that an AI can write a brilliant historical analysis yet fail to count how many times the letter e appears in a single word. This happens because the model sees the entire word as one abstract token ID rather than an ordered sequence of letters.
  • Sensitivity to formatting: Adding an accidental space before a word, capitalizing random letters, or swapping straight quotation marks for curly ones can change how the text is sliced, causing the model to interpret the prompt differently.
  • Language cost disparities: Because tokenizers are predominantly trained on common languages, English text uses tokens very efficiently. Languages with fewer online training resources or non-Latin scripts often break into tiny fragments, requiring more tokens for the exact same message.

Best practices for managing tokens

Every language model operates with a strict context window, meaning it can only hold a certain number of tokens in memory at once. Furthermore, AI service providers charge for usage based on the total volume of tokens processed. To keep operations lean, users should clean up unnecessary whitespace, fix obvious typos, and eliminate repetitive boilerplate text before submitting large batches of documents.

Frequently asked questions

Is one token equal to one word?

Not necessarily. In standard English, a common rule of thumb is that one token represents roughly three-quarters of a word, meaning 100 words typically equate to around 130 to 140 tokens. Complex terms, code, or typos usually require multiple tokens per word.

Do punctuation marks and spaces count as tokens?

Yes. Punctuation marks, special symbols, and even leading spaces often count as individual tokens or alter how adjacent words are divided by the tokenizer.

Why do typos increase token usage?

Tokenizers store common whole words in their dictionary. When you introduce a typo, the system cannot find the word and must break it into several smaller letter fragments, inflating the total token count.

← All terms in the AI Glossary

🛒 Sklep Kinetyka CapCut Pro 460 zł/rok Lovable Pro 450 zł/rok Replit Core 119 zł/rok Zobacz wszystko →
× ‹ powiększenie ›