Token Explorer · Copilot Ready

Know your token budget before you hit Send.
Visualise exactly how any Markdown or plain-text file is tokenized — live, as you type — for the AI model of your choice. See token counts, context-window usage, and estimated API cost before handing files to GitHub Copilot or any other LLM.

What's New in v0.5.0

• Request Lens — new opt-in sidebar view that reads local Copilot transcript logs and breaks each chat session down by prompt, assistant reply, reasoning, and tool-call tokens.
• Session detail webview — click a Request Lens session to see where visible tokens went, including per-request category bars, tool-call counts, response previews, and estimated agent-loop amplification.
• Request Lens commands — added commands to open a captured session, refresh the Request Lens data, and jump straight to the Request Lens setting.
• Privacy-first default — Request Lens is disabled by default; no Copilot transcript files are read unless tokenExplorer.requestLens.enabled is turned on.

What's New in v0.4.0

• CodeLens — a live lens appears at the top of every Markdown / plain-text file showing the model name, token count, context-window %, estimated input cost, and an Open Token Preview shortcut. Clicking the lens opens the model picker.
• Expanded file-type support — the Workspace Files view now scans .mdx, .rst, and .log files in addition to .md and .txt.
• Status bar context-window meter — the status-bar pill now includes an inline ASCII bar (████░░░░░░░░ 1.23%) alongside the token count. The bar turns yellow above 90% and red at 100%.
• Status bar tooltip caveats — hovering the status-bar pill shows any model-specific notes (estimated pricing, proxy tokenizer, unverified context limit).
• Model picker caveat flags — the Quick Pick for model selection now shows inline flags (! proxy tokenizer, ! est. pricing) so you know before switching which numbers are approximate.
• 2026 model refresh — expanded roster to 26 models: GPT-5.5, GPT-5.4, GPT-5.4 mini, GPT-5.3 Codex, GPT-5.2, GPT-5.2 Codex, GPT-5 mini, Claude Opus 4.7, Claude Sonnet 4.6, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3 Flash.
• Heuristic BPE adapter — new heuristicAdapter covers models without a public tokenizer (Grok, Raptor, LLaMA, Mistral, Cohere, newer GPT-5 variants) using a calibrated BPE proxy.

What's New in v0.3.1

• TypeScript fix — corrected Node.js type resolution (NodeJS.Timeout, setTimeout, clearTimeout, TextDecoder) in VS Code's language service by adding "types": ["node"] to tsconfig.json.
• Code cleanup — removed a stale legacy provider reference that was never registered but remained as dead code.

What's New in v0.3.0

Selection is now the single source of truth

Everything in the extension — the Dashboard, the status-bar pill, and the Compare Models command — now reads from one place: the set of files you have checked in the Workspace Files view.

• Dashboard — unified token budget section
  A single token budget block replaces the old separate "Context capacity" and "Selected files" panels. It shows:

  • A scope pill that always names what is being counted: N selected file(s) (cyan) or active editor (no files selected) (amber, fallback).
  • One ASCII progress bar (████░░░░) that reflects the combined token spend, colour-coded green → yellow → red as you approach or exceed the context limit.
  • Used / Free / Est. cost rows.
  • A files in scope table with a summary footer showing total context used (tokens + %ctx) and the tokens remaining for your prompt and response — the two numbers that matter most.

• Status-bar pill always shows the same number as the Dashboard: model · N tok · M file(s) when a selection is active, or model · N tok · active when falling back to the open editor.

• Compare across models runs against the selected files (with a clear fallback notice in the Output channel when nothing is selected).

• + add active file button in the Dashboard ticks the currently open editor into the Workspace Files selection without leaving the panel.

Checkbox flicker fixed

Checking or un-checking a file or folder in the Workspace Files view no longer produces a brief visual flash where the checkbox appears to toggle twice. The root cause was a redundant second render pass; this has been eliminated.

What's New in v0.2.0

• Workspace Files view — auto-discovers and tokenizes every Markdown / text file in the workspace; token count and context % shown inline next to each file name.
• Multi-file selection budget calculator — check multiple files to see combined token count, context-window %, and estimated input cost in real time.
• Separate Markdown / Text sections — files grouped by type with cascade checkboxes on category and folder nodes.
• Workspace Files toolbar — Refresh and Clear Selection buttons in the view title bar.

Features

Live Token Count in the Status Bar

A persistent status-bar item shows the token count for whatever is currently in scope:

• Selection active → ⟨model⟩ · ⟨N⟩ tok · ⟨M⟩ file(s) ████░░░░░░░░ 1.23%
• No selection → ⟨model⟩ · ⟨N⟩ tok · active ████░░░░░░░░ 1.23%

The inline ASCII bar turns yellow above 90% context usage and red at 100%. Hover for a tooltip showing the exact context breakdown and any model-specific caveats.

Activity-Bar Sidebar

A dedicated Token Explorer panel in the activity bar gives you three views:

Dashboard — Unified Token Budget

The Dashboard (v0.3) has four collapsible sections:

1. Active model — name, algorithm, vocab size, context limit, price/1M tokens, change model button. Any unverified fields show a caveat note inline.
2. Token budget — scope pill + ASCII bar + used / free / cost rows + action buttons (compare, add active file, clear selection, refresh).
3. Files in scope — table of every selected file with its token count and %ctx, plus a footer row showing the total context consumed and tokens remaining.
4. Active editor (when a supported file is open) — file path, token count, open token preview button, and an + add to selection shortcut.

Workspace Files — Multi-file Budget Calculator

1. Open a workspace — the view scans and tokenizes every Markdown, MDX, reStructuredText, log, and plain-text file in the background.
2. Each file shows ⟨N⟩ tok · ⟨X.XX⟩% alongside its name. Hover for a detailed tooltip.
3. Tick the files you plan to pass to a prompt. Tick a category or folder node to select everything inside it in one click.
4. The Dashboard and status bar update instantly to reflect the new selection.
5. Switch models — all counts and percentages update for the new model automatically.

Example: Tick four Markdown docs you want to include in a Copilot prompt. The budget section shows 18,420 tokens used — 1.84% of GPT-4.1's 1M context window, leaving 981,580 tokens free, at an estimated cost of $0.0368 input tokens.

Inline Editor Highlighting

Enable Token Explorer: Toggle Inline Highlighting to colour each token directly inside the editor using a six-colour cycling palette — ideal for understanding exactly where token boundaries fall.

Token Preview Webview

Open a side panel that renders the document as coloured token chips, one chip per token. Newlines and tabs are shown as visible glyphs (⏎ / →). Per-model caveats are listed below the chips.

Request Lens — Copilot Session Token Breakdown

Enable tokenExplorer.requestLens.enabled to inspect local Copilot transcript logs for the current workspace. The Request Lens view lists captured sessions newest-first; opening a session shows:

• Visible content total — the prompt, assistant reply, reasoning, and tool-call tokens recorded in the transcript.
• Per-request breakdowns — each request gets the same category bars so you can see which turn drove the token spend.
• Tool-call context — tool-call counts and tool names are shown when an agentic session used tools.
• Estimated amplification — long tool-using sessions show an approximate processed-token multiplier because the agent re-sends growing context across tool round trips.

Request Lens is disabled by default. When disabled, Token Explorer does not read Copilot transcript files and the view stays inactive.

26-Model Coverage

Switch models at any time from the command palette or settings. Every model in the list below is selectable:

OpenAI / GPT family — GPT-5.5, GPT-5.4, GPT-5.4 mini, GPT-5.3 Codex, GPT-5.2, GPT-5.2 Codex, GPT-5 mini, GPT-4.1, GPT-4o, GPT-3.5 Turbo
Anthropic / Claude family — Claude Opus 4.7, Claude Sonnet 4.6, Claude Sonnet 4.5, Claude Sonnet 4, Claude Haiku 4.5, Claude 3.5 Sonnet
Google / Gemini family — Gemini 3.1 Pro, Gemini 3 Flash, Gemini 2.5 Pro, Gemini 1.5 Pro
Other — Grok Code Fast 1, Raptor mini (GitHub Copilot internal), LLaMA 3 70B, LLaMA 2 70B, Mistral 7B v0.1, Cohere Command R+

Real tokenizer libraries are used wherever possible:

Accuracy caveats are shown inline whenever a proxy tokenizer or estimated pricing is used.

Copilot Chat Participant — @tokens

Copilot Agent Tools

Token Explorer registers six Language Model Tools that Copilot's default agent can call automatically — no @tokens prefix required. When you ask Copilot something like "how big is this file for Claude?" or "will my selected files fit in GPT-4o's context window?", the agent picks the right tool and answers directly.

All tools run fully offline — no API keys, no network calls.

Getting Started

1. Install the extension.
2. Open any workspace folder — the Workspace Files view immediately starts scanning and tokenizing every Markdown, MDX, RST, log, and plain-text file.
3. Open any Markdown or plain-text file for live CodeLens, status-bar updates, and the Dashboard.
4. The status bar shows the token count for the default model (gpt-4.1).

Plan a multi-file prompt budget

1. Click the Token Explorer icon in the activity bar.
2. Open the Workspace Files section.
3. Tick the files you intend to include in a prompt. Tick a category or folder to select all files inside it.
4. The Dashboard → Token budget section shows the combined total, the ASCII gauge, and the number of tokens still free for your prompt and the model's response.

Change the Active Model

• Command palette → Token Explorer: Select Model
• Settings → tokenExplorer.defaultModel
• Dashboard → change model button

All views update automatically.

Open the Token Preview

Command palette → Token Explorer: Open Token Preview
A panel opens beside your editor showing every token as a coloured chip.

Commands

Settings

Requirements

• VS Code 1.95.0 or later
• No external API keys required — all tokenizers run fully offline inside the extension

Known Limitations

• The Workspace Files view supports .md, .mdx, .markdown, .rst, .log, .txt, and .text files. Other file types (Python, TypeScript, etc.) are not scanned automatically, though the #tokenCountFile agent tool can tokenize any text file by path.
• Tokenization of very large files (>1 MB) may be slow when inline highlighting is on.
• Gemini, Grok, Raptor, and several newer GPT-5 variants use proxy or heuristic tokenizers. Token counts are approximate. Accuracy caveats are always displayed in the UI.
• Cost estimates for Copilot-bundled models (Raptor, GPT-5.x) are shown as $0 or indicative only; these models are not billed per token through the standard API.
• The Workspace Files view excludes node_modules, .git, dist, out, build, .next, and .vscode-test folders.
• Request Lens measures only what Copilot transcript logs contain. System instructions, tool definitions, tool result bodies, exact provider usage, and cache figures are not available in those logs, so Request Lens totals are a visible-content breakdown rather than a full bill.

Changelog

v0.5.0

• Request Lens — new opt-in view for local Copilot transcript analysis, showing visible token totals by session and request.
• Session detail webview — category bars for prompts, assistant replies, reasoning, and tool calls, with previews and tool-call metadata.
• Agent-loop estimate — tool-heavy sessions show approximate processed-token amplification from repeated context round trips.
• Request Lens commands and setting — added open-session, refresh, and settings commands plus tokenExplorer.requestLens.enabled, disabled by default.

v0.4.0

• CodeLens — live token count, context %, cost, and Open Token Preview lens at the top of every Markdown / plaintext file.
• Expanded file types — Workspace Files view now includes .mdx, .rst, .log in addition to .md and .txt.
• Status bar context meter — inline ASCII bar + percentage in the pill, color-coded yellow >90%, red ≥100%.
• Status bar tooltip caveats — model-specific notes shown on hover.
• Model picker caveat flags — Quick Pick shows ! proxy tokenizer / ! est. pricing / ! est. context flags per model.
• 26-model roster — GPT-5.5, GPT-5.4, GPT-5.4 mini, GPT-5.3 Codex, GPT-5.2, GPT-5.2 Codex, GPT-5 mini, Claude Opus 4.7, Claude Sonnet 4.6, Claude Sonnet 4.5, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3 Flash.
• Heuristic BPE adapter — calibrated proxy for models without a public tokenizer (Grok, Raptor, LLaMA, Mistral, Cohere, newer GPT-5 variants).
• Inline accuracy caveats — per-model notes for unverified pricing, context limits, and proxy tokenizers surfaced in the Dashboard, status bar, and model picker.

v0.3.1

• TypeScript fix — added "types": ["node"] to tsconfig.json so VS Code's language service correctly resolves Node.js globals (NodeJS.Timeout, setTimeout, clearTimeout, TextDecoder). The extension compiled fine from the command line, but the VS Code Problems panel reported false errors.
• Code cleanup — removed a dead legacy DashboardTreeProvider import and void reference that was left over after the dashboard was migrated to a webview.

v0.3.0

• Selection as single source of truth — Dashboard, status bar, and compare command all read from the same checked-file set; no more ambiguity between "single-file" and "multi-file" numbers.
• Unified token budget section — replaces two separate capacity / files panels with one block: scope pill, ASCII bar, used/free/cost rows, and a files table with a totals footer.
• Files table footer — shows total tokens used and tokens remaining (with %ctx) so you can see at a glance how much room is left for the prompt and model response.
• Status-bar scope label — pill now appends · N file(s) or · active so the scope is always visible without opening the Dashboard.
• + add active file — one-click button in Dashboard to tick the open editor into the selection.
• Checkbox flicker fixed — eliminated the double-render that caused a brief visual flash when checking / unchecking items in the Workspace Files view.
• Compare across models now operates on the selection (with fallback notice if nothing is selected).
• Registered new command Token Explorer: Add Active File to Selection.

v0.2.0

• New: Workspace Files view — auto-discover and tokenize every Markdown / text file in the workspace; token count and context % shown inline next to each file name.
• New: Multi-file selection budget calculator — check multiple files to see combined token count, context-window %, and estimated input cost in real time.
• New: Separate Markdown / Text sections — files are grouped by type in the Workspace Files view.
• New: Workspace Files toolbar buttons — Refresh and Clear Selection in the view title bar.
• Activation now triggers on onStartupFinished so the Workspace Files view populates immediately on any workspace open.

v0.1.0

• Initial release: live status-bar token count, Dashboard view, Tokens view, inline highlighting, Token Preview webview, 26-model coverage, and @tokens Copilot chat participant.

License

MIT