Tokenization is at the heart of much weirdness of LLMs. Do not brush it off.

• Why can't LLM spell words? Tokenization.
• Why can't LLM do super simple string processing tasks like reversing a string? Tokenization.
• Why is LLM worse at non-English languages (e.g. Japanese)? Tokenization.
• Why is LLM bad at simple arithmetic? Tokenization.
• Why did GPT-2 have more than necessary trouble coding in Python? Tokenization.
• Why did my LLM abruptly halt when it sees the string "<|endoftext|>"? Tokenization.
• What is this weird warning I get about a "trailing whitespace"? Tokenization.
• Why the LLM break if I ask it about "SolidGoldMagikarp"? Tokenization.
• Why should I prefer to use YAML over JSON with LLMs? Tokenization.
• Why is LLM not actually end-to-end language modeling? Tokenization.
• What is the real root of suffering? Tokenization.

Good tokenization web app: https://tiktokenizer.vercel.app

what is tokenization?

Tokenization in natural language processing splits raw text into smaller pieces across sentence-level, word-level, subword-level, and character-level granularities

what is byte level tokenization? and why it is ?

Byte-level tokenization treats text as a sequence of raw bytes (0 to 255) rather than characters. It is the core mechanism used by modern Large Language Models (LLMs) like GPT-4, Llama, and Mistral to convert text into numbers.

Why Byte-Level Tokenization Matters

• No Unknown Tokens: Standard character tokenizers fail when they meet a rare emoji, math symbol, or foreign character, resulting in an error or an "unknown" (<unk>) token. Byte-level tokenizers can read any string because every character on the internet decomposes into standard bytes.