AI Token Boundary Colorizer
See BPE token boundaries and statistics locally in your browser
Tokenize text
Tokenizer boundaries are invisible in plain text but they decide cost, context limits, and even model behavior. This colorizer encodes your text with the OpenAI BPE tokenizers (o200k_base for GPT-4o-class models, cl100k_base for GPT-4/3.5) and renders every token with its own background color, plus token IDs from the vocabulary. Statistics show total tokens, characters, and the characters-per-token ratio so you can estimate prompt costs and see why unusual spellings or code fragments tokenize inefficiently. Tokenizer tables load lazily; all encoding happens locally and nothing is uploaded.
Related guides
Related tools
Agent Tool Flowchart Builder
Design agentic tool-call DAGs with decision branches and bounded loops. Validates reachability, cycle handling, and emits clean Mermaid flowcharts — fully in your browser.
AI Agent Memory & Buffer Simulator
Compare sliding window, summary buffer, and vector memory strategies
AI Code Explainer
Explain code snippets with grounded cloud AI
AI Commit Message Generator
Suggest Conventional Commits messages from a git diff using on-device WebLLM
AI Documentation Generator
Generate validated JSDoc/TSDoc blocks with cloud AI
AI Regex Generator
Turn natural language into a regex pattern with cloud AI