Token Explorer · Copilot Ready

Know your token budget before you hit Send.
Visualise exactly how any Markdown or plain-text file is tokenized — live, as you type — for the AI model of your choice. See token counts, context-window usage, and estimated API cost before handing files to GitHub Copilot or any other LLM.

What's New in v0.3.1

• TypeScript fix — corrected Node.js type resolution (NodeJS.Timeout, setTimeout, clearTimeout, TextDecoder) in VS Code's language service by adding "types": ["node"] to tsconfig.json.
• Code cleanup — removed a stale legacy provider reference that was never registered but remained as dead code.

What's New in v0.3.0

Selection is now the single source of truth

Everything in the extension — the Dashboard, the status-bar pill, and the Compare Models command — now reads from one place: the set of files you have checked in the Workspace Files view.

• Dashboard — unified token budget section
  A single token budget block replaces the old separate "Context capacity" and "Selected files" panels. It shows:

  • A scope pill that always names what is being counted: N selected file(s) (cyan) or active editor (no files selected) (amber, fallback).
  • One ASCII progress bar (████░░░░) that reflects the combined token spend, colour-coded green → yellow → red as you approach or exceed the context limit.
  • Used / Free / Est. cost rows.
  • A files in scope table with a summary footer showing total context used (tokens + %ctx) and the tokens remaining for your prompt and response — the two numbers that matter most.

• Status-bar pill always shows the same number as the Dashboard: model · N tok · M file(s) when a selection is active, or model · N tok · active when falling back to the open editor.

• Compare across models runs against the selected files (with a clear fallback notice in the Output channel when nothing is selected).

• + add active file button in the Dashboard ticks the currently open editor into the Workspace Files selection without leaving the panel.

Checkbox flicker fixed

Checking or un-checking a file or folder in the Workspace Files view no longer produces a brief visual flash where the checkbox appears to toggle twice. The root cause was a redundant second render pass; this has been eliminated.

What's New in v0.2.0

• Workspace Files view — auto-discovers and tokenizes every Markdown / text file in the workspace; token count and context % shown inline next to each file name.
• Multi-file selection budget calculator — check multiple files to see combined token count, context-window %, and estimated input cost in real time.
• Separate Markdown / Text sections — files grouped by type with cascade checkboxes on category and folder nodes.
• Workspace Files toolbar — Refresh and Clear Selection buttons in the view title bar.

Features

Live Token Count in the Status Bar

A persistent status-bar item shows the token count for whatever is currently in scope:

• Selection active → ⟨model⟩ · ⟨N⟩ tok · ⟨M⟩ file(s)
• No selection → ⟨model⟩ · ⟨N⟩ tok · active

Hover the pill for a tooltip that explains exactly what is being counted.

Activity-Bar Sidebar

A dedicated Token Explorer panel in the activity bar gives you three views:

Dashboard — Unified Token Budget

The Dashboard (v0.3) has four collapsible sections:

1. Active model — name, algorithm, vocab size, context limit, price/1M tokens, change model and sync copilot buttons.
2. Token budget — scope pill + ASCII bar + used / free / cost rows + action buttons (compare, add active file, clear selection, refresh).
3. Files in scope — table of every selected file with its token count and %ctx, plus a footer row showing the total context consumed and tokens remaining.
4. Active editor (when a supported file is open) — file path, token count, open token preview button, and an + add to selection shortcut.

Workspace Files — Multi-file Budget Calculator

1. Open a workspace — the view scans and tokenizes every Markdown and text file in the background.
2. Each file shows ⟨N⟩ tok · ⟨X.XX⟩% alongside its name. Hover for a detailed tooltip.
3. Tick the files you plan to pass to a prompt. Tick a category or folder node to select everything inside it in one click.
4. The Dashboard and status bar update instantly to reflect the new selection.
5. Switch models — all counts and percentages update for the new model automatically.

Example: Tick four Markdown docs you want to include in a Copilot prompt. The budget section shows 18,420 tokens used — 1.84% of GPT-4.1's 1M context window, leaving 981,580 tokens free, at an estimated cost of $0.0368 input tokens.

Inline Editor Highlighting

Enable Token Explorer: Toggle Inline Highlighting to colour each token directly inside the editor using a six-colour cycling palette — ideal for understanding exactly where token boundaries fall.

Token Preview Webview

Open a side panel that renders the document as coloured token chips, one chip per token. Newlines and tabs are shown as visible glyphs (⏎ / →). Per-model caveats are listed below the chips.

Sync with Copilot Chat Model

Click sync copilot in the Dashboard (or run Token Explorer: Sync Model with Copilot Chat) to have the extension pick the best-matching model from your available Copilot Chat models automatically.

26-Model Coverage

Switch models at any time from the command palette or settings. Every model in the list below is selectable:

OpenAI / GPT family — GPT-5.5, GPT-5.4, GPT-5.4 mini, GPT-5.3 Codex, GPT-5.2, GPT-5.2 Codex, GPT-5 mini, GPT-4.1, GPT-4o, GPT-3.5 Turbo
Anthropic / Claude family — Claude Opus 4.7, Claude Sonnet 4.6, Claude Sonnet 4.5, Claude Sonnet 4, Claude Haiku 4.5, Claude 3.5 Sonnet
Google / Gemini family — Gemini 3.1 Pro, Gemini 3 Flash, Gemini 2.5 Pro, Gemini 1.5 Pro
Other — Grok Code Fast 1, Raptor mini (GitHub Copilot internal), LLaMA 3 70B, LLaMA 2 70B, Mistral 7B v0.1, Cohere Command R+

Real tokenizer libraries are used wherever possible:

Accuracy caveats are shown inline whenever a proxy tokenizer or estimated pricing is used.

Copilot Chat Participant — @tokens

Copilot Agent Tools

Token Explorer registers six Language Model Tools that Copilot's default agent can call automatically — no @tokens prefix required. When you ask Copilot something like "how big is this file for Claude?" or "will my selected files fit in GPT-4o's context window?", the agent picks the right tool and answers directly.

All tools run fully offline — no API keys, no network calls.

Getting Started

1. Install the extension.
2. Open any workspace folder — the Workspace Files view immediately starts scanning and tokenizing every Markdown and text file.
3. Open any Markdown or plain-text file for live status-bar updates and the Dashboard.
4. The status bar shows the token count for the default model (gpt-4.1).

Plan a multi-file prompt budget

1. Click the Token Explorer icon in the activity bar.
2. Open the Workspace Files section.
3. Tick the files you intend to include in a prompt. Tick a category or folder to select all files inside it.
4. The Dashboard → Token budget section shows the combined total, the ASCII gauge, and the number of tokens still free for your prompt and the model's response.

Change the Active Model

• Command palette → Token Explorer: Select Model
• Settings → tokenExplorer.defaultModel
• Dashboard → change model button
• Dashboard → sync copilot button (matches your Copilot Chat selection)

All views update automatically.

Open the Token Preview

Command palette → Token Explorer: Open Token Preview
A panel opens beside your editor showing every token as a coloured chip.

Commands

Settings

Requirements

• VS Code 1.95.0 or later
• No external API keys required — all tokenizers run fully offline inside the extension

Known Limitations

• Only Markdown and plain-text files are supported in the Workspace Files view. Support for additional languages (Python, TypeScript, etc.) is planned.
• Tokenization of very large files (>1 MB) may be slow when inline highlighting is on.
• Gemini, Grok, Raptor, and several newer GPT-5 variants use proxy or heuristic tokenizers. Token counts are approximate. Accuracy caveats are always displayed in the UI.
• Cost estimates for Copilot-bundled models (Raptor, GPT-5.x) are shown as $0 or indicative only; these models are not billed per token through the standard API.
• The Workspace Files view excludes node_modules, .git, dist, out, build, .next, and .vscode-test folders.
• There is no public VS Code API to read the model currently active in Copilot Chat; sync copilot uses a best-effort heuristic match.

Changelog

v0.3.1

• TypeScript fix — added "types": ["node"] to tsconfig.json so VS Code's language service correctly resolves Node.js globals (NodeJS.Timeout, setTimeout, clearTimeout, TextDecoder). The extension compiled fine from the command line, but the VS Code Problems panel reported false errors.
• Code cleanup — removed a dead legacy DashboardTreeProvider import and void reference that was left over after the dashboard was migrated to a webview.

v0.3.0

• Selection as single source of truth — Dashboard, status bar, and compare command all read from the same checked-file set; no more ambiguity between "single-file" and "multi-file" numbers.
• Unified token budget section — replaces two separate capacity / files panels with one block: scope pill, ASCII bar, used/free/cost rows, and a files table with a totals footer.
• Files table footer — shows total tokens used and tokens remaining (with %ctx) so you can see at a glance how much room is left for the prompt and model response.
• Status-bar scope label — pill now appends · N file(s) or · active so the scope is always visible without opening the Dashboard.
• + add active file — one-click button in Dashboard to tick the open editor into the selection.
• Checkbox flicker fixed — eliminated the double-render that caused a brief visual flash when checking / unchecking items in the Workspace Files view.
• Compare across models now operates on the selection (with fallback notice if nothing is selected).
• Registered new command Token Explorer: Add Active File to Selection.

v0.2.0

• New: Workspace Files view — auto-discover and tokenize every Markdown / text file in the workspace; token count and context % shown inline next to each file name.
• New: Multi-file selection budget calculator — check multiple files to see combined token count, context-window %, and estimated input cost in real time.
• New: Separate Markdown / Text sections — files are grouped by type in the Workspace Files view.
• New: Workspace Files toolbar buttons — Refresh and Clear Selection in the view title bar.
• Activation now triggers on onStartupFinished so the Workspace Files view populates immediately on any workspace open.

v0.1.0

• Initial release: live status-bar token count, Dashboard view, Tokens view, inline highlighting, Token Preview webview, 26-model coverage, and @tokens Copilot chat participant.

License

MIT