LLM Tokenizer

The ultimate AI token counter for your IDE.
Token counting for 71 models — exact where the tokenizer is public, honestly labelled where it is not.

Optimized for developers building with LLMs.

LLM Tokenizer gives you instant visibility into your token usage directly within your IDE. Whether you're optimizing prompts, estimating API costs, or ensuring your context window limits aren't exceeded, LLM Tokenizer removes the guesswork.

• Check Context Limits: Know instantly if your file fits within the context window of your favorite AI model.
• Estimate Costs: Get a clear sense of input token volume before sending requests to expensive APIs.
• Optimize RAG Pipelines: Analyze folder-level token counts to better chunk your knowledge base.

Stop copying and pasting into web calculators. Get precise counts right where you code.

Features

🎯 Core Features

• Exact counts, not guesses: 52 of the 71 supported models are tokenized with the model's own tokenizer. The rest are clearly marked with ≈.
• Real-time Token Count: The active file's token count in the Status Bar
• Context Limit Warnings: Indicators at 80% and 100% of the model's usable input limit
• Project-wide Counting: Workspace totals with caching, cancellation, and multi-root support
• Multi-file Selection: Select multiple files or folders in the explorer for a batch count
• Folder Analysis: Right-click a folder to count recursively
• Selection Counting: Count only the text you highlighted
• Runs off the UI thread: Tokenizing happens in a worker thread, so the editor never blocks
• Persistent Preferences: Your model choice is remembered

🔒 Privacy

Your code never leaves your machine. There is no telemetry and no network request that contains file contents. The only network access is a one-time download of a model's vocabulary file from huggingface.co, which you can turn off with llm-tokenizer.downloadTokenizers.

⚙️ Configuration

• llm-tokenizer.defaultModel: Model used until you pick one
• llm-tokenizer.statusBarDisplay: "file", "project", or "both"
• llm-tokenizer.ignoreGitignoredFiles: Exclude gitignored files from folder and workspace totals
• llm-tokenizer.enableProjectScan: Turn off workspace-wide counting on very large repositories
• llm-tokenizer.downloadTokenizers: Allow the one-time tokenizer download that makes counts exact

Supported Models

71 models across 13 providers. Every id is checked against the provider's own documentation; models that a provider has retired are removed, and your setting is migrated automatically.

¹ after a one-time tokenizer download ² these two releases are too recent for Google's SDK to map them to a published vocabulary, so they fall back to a character estimate

Usage

Basic Operations

1. Open a file: Token count appears in Status Bar (bottom right)
2. Click Status Bar item to change model
3. Right-click a single file → Count Tokens (shows a popup notification with the token count)
4. Right-click a folder (or multiple files) → Count Tokens (opens a summary showing where the tokens are, by folder and by language)
5. Select text in editor → Count Tokens to count only the selection

Configuration

Open Settings (Ctrl/Cmd+,) and search for "LLM Tokenizer":

• Default Model: the model used until you pick one from the status bar
• Status Bar Display: "file", "project", or "both" (default "both")
• Ignore Gitignored Files: exclude .gitignore matches from counts (on by default)
• Enable Project Scan: turn off workspace-wide counting on very large repositories
• Download Tokenizers: allow the one-time download that makes counts exact

Context Warnings

• Normal: under 80% of the model's usable input limit
• Warning: 80–99%
• Error: at or over 100%

A leading ≈ means the count is an estimate rather than an exact tokenization.

Accuracy

There are three tiers, and the status bar tells you which one you are in. Two vocabulary formats are published in the wild — a Hugging Face tokenizer.json, and a bare tiktoken rank table (Moonshot ships the latter) — and both are read.

Why some models are only estimated. Anthropic and xAI do not publish a tokenizer for any current model, and Anthropic explicitly advises against approximating Claude with OpenAI's tokenizer. An honest estimate is better than a confident wrong number, so those models are marked rather than dressed up.

The per-family ratios are measured against real corpora where a tokenizer exists to measure against. Grok is the exception: no Grok tokenizer has ever been published, so its ratio is inferred rather than measured, and it is the least reliable number here.

Known limitation: only the .gitignore at the root of each workspace folder is read, plus .git/info/exclude. Nested .gitignore files deeper in the tree are not applied.

Earlier versions approximated everything with cl100k_base and a fudge factor. Measured against the real tokenizers, that undercounted Gemini on JSON by 27% and Mistral on source code by 24% — errors in the direction that tells you your prompt fits when it does not.

Requirements

VS Code 1.105.0+

Changelog

See CHANGELOG.md for detailed release notes.

Author: Matteo Teodori