The "gpt-token-counter-live" is a Visual Studio Code extension that displays the token count of selected text or the entire open document in the status bar. The token count is determined per model family using: GPT via tiktoken, Claude via Anthropic's tokenizer, Gemini via a local approximation, and any HuggingFace tokenizer via @huggingface/tokenizers (Qwen, Llama, Mistral, etc.).
NEW in v1.5.0: Count and highlight tokens for any HuggingFace tokenizer (Qwen, Llama, Mistral, and more), loaded from the Hub or a local tokenizer.json. Plus file pattern filtering to scope the counter to specific file types.
This tool is built to get a speedy token counting result right on VS Code while you are working on prompting files. I personally needed a lot while working on many LLM projects, so I decided to make one for myself. I hope this helps you too!
Features
Real-Time Token Counting
Live token counting for the current selection or entire document, displayed directly in the status bar. Counts update automatically as you type or change your selection.
Multi-Model Family Support
Click the status bar to switch between model families: GPT (OpenAI), Claude (Anthropic), or Gemini (Google AI).
- GPT (OpenAI): Uses tiktoken
encoding_for_model('gpt-5') with fallbacks to o200k_base → cl100k_base for accurate token counting across all GPT models.
- Claude (Anthropic): Uses Anthropic's official tokenizer for precise token boundaries with full highlighting support.
- Gemini (Google AI): Approximates tokens using GPT encodings or ~4 chars/token fallback (highlighting not available).
- HuggingFace: Loads a
tokenizer.json from the Hub (or a local file) and reuses it for live counts. Byte-level BPE tokenizers (Qwen, Llama, Mistral, etc.) support highlighting; SentencePiece tokenizers that alter whitespace fall back to counting only.
Visual Token Highlighting
See your tokens in real-time with alternating color bands that show exactly where each token begins and ends. Available for GPT, Claude, and HuggingFace byte-level BPE tokenizers (Qwen, Llama, Mistral, etc.).
Key features:
- Toggle on/off: Click the palette icon in the status bar to enable/disable highlighting
- Smart text contrast: Foreground text color automatically adapts to your highlight colors for optimal readability
- Customizable colors: Choose your own colors with full alpha/transparency support
- Editor-aware: Only highlights in text editors; Output/Debug panes remain clean
Customizable Highlight Colors
Open the Command Palette and run Configure Token Highlight Colors to access a dedicated color configurator.
Features:
- Separate color pickers for even/odd token bands
- Hex color input with opacity sliders
- Live preview showing exactly how colors will look
- Smart contrast preview so you can ensure text remains readable
Customizable Status Bar Display
Personalize how token information appears in your status bar using template placeholders.
Supported placeholders:
{count} - Token count{family} or {model} - Model family name (GPT, Claude, Gemini){provider} - Provider name (openai, anthropic, gemini)
Requirements
- Visual Studio Code: The extension is developed for VS Code and will not work with other editors.
Commands
This extension provides the following commands (accessible via Command Palette):
Change Model Family: Switch between GPT (OpenAI), Claude (Anthropic), and Gemini (Google AI) tokenizers. Also accessible by clicking the token count in the status bar.
Toggle Token Highlighting: Enable or disable visual token highlighting overlays. Also accessible by clicking the palette icon in the status bar.
Configure Token Highlight Colors: Open an interactive color configurator to customize the highlight colors for even and odd token bands. Includes live preview and smart text contrast.
Count Tokens: Manually trigger token counting for the current document or selection.
Extension Settings
This extension contributes the following settings:
Model & Display Settings
gpt-token-counter-live.defaultModelFamily: Choose which model family activates by default when you open VS Code.
- Options:
openai, anthropic, gemini, or huggingface
- Default:
openai
gpt-token-counter-live.statusBarDisplayTemplate: Customize how token information appears in the status bar.
- Default:
Token Count: {count} ({family})
- Supported placeholders:
{count}, {family}, {model} (alias for family), {provider}
gpt-token-counter-live.enabledFilePatterns: Glob patterns for files where the status bar should be shown.
- Default:
[] (empty array shows for all files)
- Example:
["*.md", "*.mdc"] shows only for markdown files
HuggingFace Tokenizers
Select HuggingFace (huggingface) in the model family picker, then point the extension at a tokenizer with either of the settings below. The first load fetches tokenizer.json from the Hub and caches it under the extension's global storage directory, so subsequent launches are offline-friendly.
gpt-token-counter-live.huggingfaceModelId: HuggingFace repo ID (e.g. Qwen/Qwen2.5-7B-Instruct, meta-llama/Llama-3-8B, mistralai/Mistral-7B-Instruct-v0.3). The extension fetches https://huggingface.co/{id}/resolve/main/tokenizer.json plus tokenizer_config.json and caches them on disk.gpt-token-counter-live.huggingfaceTokenizerPath: Absolute path to a local tokenizer.json file. When set, it overrides huggingfaceModelId. A sibling tokenizer_config.json will be picked up automatically if present.
What works and what doesn't:
- Byte-level BPE tokenizers (Qwen, Llama, Mistral, most GPT-style tokenizers on the Hub) give precise token counts and highlighting.
- SentencePiece / Unigram tokenizers that strip or transform whitespace (T5, some BERT variants) still give precise counts, but highlighting is automatically disabled. Decoding an id sequence back to text doesn't produce an identical string, so offsets can't be attributed reliably. You'll see token counts but no color overlays.
- Gated or private repositories cannot be fetched without auth. The first load will surface the raw HTTP status in a VS Code error notification (typically
401 or 403); download tokenizer.json manually and point huggingfaceTokenizerPath at it.
- Missing repositories / offline first-load show the same error notification and fall back to a rough
~4 chars/token approximation until the tokenizer resolves.
Highlighting Configuration
Token highlight colors are stored in your VS Code global state (synced across devices if you have Settings Sync enabled). To customize them select Configure Token Highlight Colors option from the Command Palette.
Quick toggle: Click the palette icon in the status bar to enable/disable token highlighting instantly.
Known Issues
There are currently no known issues. If you encounter a problem, please report it on the issue tracker.
Release Notes
1.5.0 - HuggingFace Tokenizers & File Pattern Filtering
New features:
- HuggingFace tokenizer family: load any
tokenizer.json from the Hub (first launch fetches + caches) or a local file. Byte-level BPE tokenizers (Qwen, Llama, Mistral, etc.) support highlighting; SentencePiece tokenizers that transform whitespace count tokens but skip highlights. Configure via huggingfaceModelId or huggingfaceTokenizerPath.
- New setting
enabledFilePatterns: Show status bar only for files matching specific glob patterns (e.g., ["*.md", "*.mdc"]). Empty array shows for all files.
Fixes:
- Token highlighting stays aligned around multi-byte UTF-8 characters (emoji, CJK). The renderer now matches raw bytes against the source instead of decoded token strings, so a token that splits a multi-byte character still lands on the right boundary.
- Claude highlighting works on documents containing characters that NFKC-normalize to a different form (e.g., full-width
（） becoming ASCII ()). Token ranges are reprojected from the normalized tokenization back onto the original text.
1.4.0 - Token Highlighting & Customization
Major new features:
Visual Token Highlighting: See exactly where each token begins and ends with alternating color bands overlaid on your text
- Available for GPT (OpenAI) and Claude (Anthropic) tokenizers
- Smart text contrast automatically adjusts foreground color for readability
- Editor-aware: only applies to text editors, keeps Output/Debug panes clean
Interactive Color Configurator: New command Configure Token Highlight Colors with:
- Separate color pickers for even and odd token bands
- Hex color input with opacity sliders for full alpha support
- Real-time preview showing exactly how colors will appear
- Smart contrast preview ensures text remains readable
Status Bar Palette Toggle: Quick access toggle button in status bar
- Click to instantly enable/disable token highlighting
- Visual states: Active (on), Inactive (off), Unavailable (for unsupported models)
New Configuration Settings:
defaultModelFamily: Choose which model family (GPT, Claude, or Gemini) activates by defaultstatusBarDisplayTemplate: Customize status bar text with template placeholders like {count}, {family}, {provider}
Technical improvements:
- Better Unicode normalization handling for Claude tokenizer (NFKC)
- Performance optimizations for real-time highlighting
- Improved error handling and user feedback
1.3.0
- Switch to model families in the UI: GPT, Claude, Gemini.
- Add Gemini token counting (approximate:
o200k_base/cl100k_base, fallback ~4 chars/token).
- GPT tokenizer now uses
encoding_for_model('gpt-5') with graceful fallbacks.
- Updated
tiktoken to 1.0.22.
1.2.3
- Added support for new OpenAI models: o3-mini, o1, o1-mini, gpt-4o-mini
- Updated to tiktoken 1.0.20
- Updated Claude models to only include Claude-3.5, Claude-3.7
- Removed older models: text-davinci-003, davinci, babbage
- Removed Claude-2 and Claude-3
1.2.1
- Moved from
gpt-tokenizer to tiktoken package.
- Fixed the special tokens issue.
1.2.0
- Modified the code to increase security.
- Added support for GPT-4o tokenizer.
- Removed unused models from the tokenizer list.
- Added Claude-3 as option using approximate token count.
1.1.0
- Added support for Claude tokenizer.
1.0.0
- Initial release of gpt-token-counter-live.
- Provides a token count in the status bar for the selected text or the entire document.
- Automatically updates the token count as text is edited or selected.
- Allows the user to select the model to use for token counting.
