vscode-folder-token-estimator

Live token estimation badges in VS Code Explorer for files, folders, and workspace root totals.

This extension helps you quickly understand context size while navigating codebases, so you can decide what to include in prompts before copying files or sending large chunks to LLM tools.

Important Disclaimer

Token counts shown by this extension are approximate estimates, not exact tokenizer outputs.

• Different models use different tokenizers.
• The same text can produce different token counts across providers and model versions.
• Billing and context accounting can vary by model family and platform.

Use these badges for fast relative sizing, not exact cost prediction or hard token-limit guarantees.

Overview

vscode-folder-token-estimator decorates Explorer entries with estimated token counts.

• File badges show per-file estimates.
• Folder badges show recursive totals of descendant files.
• Workspace root badges show repository-level totals.
• Tooltips always show exact numeric totals.

The extension uses a fast, model-agnostic character heuristic by default and is designed for low-latency updates during normal editing.

Table of Contents

• Features
• Architecture
• Token Estimation Logic
• Requirements
• Installation
• Quick Start
• Screenshots
• Commands
• Configuration
• Badge Legend
• Performance Notes
• Troubleshooting
• Development
• Contributing
• Security and Privacy

Features

• Live Explorer decorations for files, folders, and repo roots.
• Incremental updates on file saves and file-system changes.
• Debounced full rescans for bulk operations.
• Exclusion support through configurable glob patterns.
• Binary file detection and large-file skipping for speed.
• Toggle and manual refresh commands.
• Compact badge rendering with exact values in tooltip.

Architecture

The extension is built around a FileDecorationProvider that maintains two in-memory maps.

• fileTokenMap: per-file estimates.
• folderTokenMap: aggregated folder totals.

Update flow:

1. A file estimate changes.
2. Delta is computed against the old value.
3. Delta is propagated to each ancestor folder up to workspace root.
4. Explorer is notified to repaint decorations.

This delta strategy avoids expensive full subtree recomputation on every edit.

Token Estimation Logic

Current estimate is intentionally simple and fast:

1. Read UTF-8 file content.
2. Normalize line endings to LF.
3. Compute estimate using:

tokens = ceil(character_count / charsPerToken)

Default charsPerToken is 4, configurable via settings.

This heuristic is model-agnostic and intentionally approximate. Expect variation versus exact tokenizers used by specific models for context limits and pricing.

Notes:

• Empty files evaluate to 0.
• Files larger than maxFileSizeKb are skipped.
• Binary-like files are skipped when null bytes are detected.

Requirements

• VS Code 1.90.0 or newer.
• A local folder or multi-root workspace opened in VS Code.

Installation

From source

1. Clone this repository.
2. Install dependencies.
3. Build extension.

npm install
npm run compile

Quick Start

1. Open this project in VS Code.
2. Press F5 to launch Extension Development Host.
3. Open any repository in the Dev Host window.
4. Observe token badges in Explorer.
5. Run Token Estimator: Refresh if you want a full recalculation.

Screenshots

Explorer Overview

Large Repo Top-Level Totals

Zero-Token Folder Decorations

Commands

Configuration

All settings live under the vscode-folder-token-estimator namespace.

Example workspace settings:

{
	"vscode-folder-token-estimator.enabled": true,
	"vscode-folder-token-estimator.charsPerToken": 4,
	"vscode-folder-token-estimator.maxFileSizeKb": 1024,
	"vscode-folder-token-estimator.excludeGlobs": [
		"**/node_modules/**",
		"**/.git/**",
		"**/dist/**",
		"**/build/**",
		"**/.venv/**",
		"**/coverage/**"
	]
}

Badge Legend

Badge text is compact for Explorer readability.

• 0 to 99: exact value.
• 100 to 999: 1h to 9h (hundreds scale).
• 1k to 999k: 1k to 9k.
• 1m and above: 1m to 9m.

Exact totals are always available in tooltip.

Performance Notes

• File-system watcher events are batched before refresh.
• Full rebuilds are debounced to avoid repeated scans.
• Large and binary files are skipped by design.
• For very large monorepos, tune excludeGlobs and maxFileSizeKb.

Troubleshooting

No badges visible

• Verify Explorer decoration badges are enabled in VS Code settings.
• Run Token Estimator: Toggle to ensure feature is enabled.
• Run Token Estimator: Refresh.
• Restart Extension Development Host after major changes.

Top-level folders missing badges

• Ensure badge rendering is enabled in Explorer settings.
• Check that folder is not excluded by excludeGlobs.
• Refresh estimates after changing exclusion settings.

Counts look different from model tokenizer output

• This extension uses a heuristic estimate, not model-specific tokenization.
• Reduce charsPerToken for a stricter estimate, or raise it for a looser one.

Development

Scripts

npm run compile
npm run watch

Run locally

1. Start watch mode or let launch pre-task handle it.
2. Press F5.
3. Use the Dev Host to validate decorations and commands.

Contributing

Contributions are welcome.

1. Fork repository.
2. Create feature branch.
3. Make focused changes with tests or validation notes.
4. Open pull request with clear before and after behavior.

Suggested contribution areas:

• Optional model-accurate tokenizer integrations.
• Top-N files view for largest token contributors.
• Performance diagnostics command.
• Workspace cache persistence between sessions.

Security and Privacy

• No remote service calls are required for estimation.
• Processing happens on local files in your workspace.
• Excluded paths can be configured to avoid scanning sensitive directories.