What is Token?

Full Definition

Tokens are the atomic units that language models read and write. A tokenizer splits raw text into tokens before feeding it to the model; common tokenization schemes (like BPE — Byte Pair Encoding) produce tokens that are roughly 3-4 characters on average in English, meaning one word is often 1-2 tokens. A model's context window is measured in tokens: GPT-4o supports 128k tokens, while some models support up to 1 million. Pricing for API access is denominated in tokens (e.g., per million input or output tokens). Understanding tokens is important for managing costs, staying within context limits, and interpreting rate-limit messages.