LLM Token Counting Cheat Sheet
What tokens are, tokenization, counting methods, context windows, and cost estimation for LLMs.
Reference
llm
tokens
tokenization
LLMs process text as tokens — subword units that are neither characters nor words. Token count drives context-window limits and API cost, so estimating it accurately matters for both.
What is a token?
Table
| Rule of thumb | Example |
|---|---|
| ~4 chars ≈ 1 token (English) | "hello world" ≈ 2-3 tokens. |
| Common words = 1 token | "the", "and". |
| Rare words split into subwords | "tokenization" → "token" + "ization". |
| Non-English often costs more | CJK chars ≈ 1-2 tokens each. |
Counting methods
Table
| Method | Notes |
|---|---|
tiktoken (OpenAI) | Official BPE tokenizer for GPT models. |
| Model-specific tokenizers | Each model family has its own vocabulary. |
| Character estimate | Rough: chars / 4. |
python
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
len(enc.encode("Hello, world!")) # token count
Context window
Table
| Model family | Approx context |
|---|---|
| GPT-4o | 128k tokens |
| Claude 3.5 Sonnet | 200k tokens |
| Gemini 1.5 Pro | 1M+ tokens |
The context window includes the system prompt, history, and output — reserve room for the completion.
Cost estimation
code
cost = (input_tokens × input_price) + (output_tokens × output_price)
Track both input and output separately, since output tokens are usually priced higher.