Local Token Counter

Local Token Counter is a Cursor/VS Code extension that counts tokens for the active editor or current selection using OpenAI-compatible tokenizers.

Marketplace Summary

Use Local Token Counter when you want a quick, local estimate of how large a file or selection is for OpenAI-compatible tokenizer budgets. It is designed for developers working with prompts, AI codebases, Markdown docs, fixtures, and tokenizer-related source files.

Highlights:

• Local-only counting in the extension host.
• Status bar count for the active file or selected text.
• Fixed encoding and model-source picker.
• Conservative skips for oversized or binary-like content.
• No telemetry, remote logging, or API-backed token counting.

The first version is intentionally local-only:

• File contents are read from the editor extension host.
• Token counts are computed locally with tiktoken.
• This extension does not transmit file contents, selections, telemetry, or usage data.
• Non-file URI schemes such as git: and virtual documents are skipped by default.

Cursor or VS Code may have unrelated features with their own data handling. This privacy statement describes only Local Token Counter.

Runtime Posture

Local Token Counter's runtime design is intentionally narrow:

• It reads text from the active editor or explicit selections through VS Code APIs.
• It tokenizes that text locally in the extension host.
• It does not execute workspace code, run shell commands, write files, create output logs, or send file content over the network.
• Release checks scan the source for forbidden network, logging, environment, and dynamic-code APIs so this posture does not drift silently.

Why Counts May Differ From Cursor

Cursor does not expose a single stable tokenizer for extensions. Actual Cursor requests can include hidden context such as rules, chat history, neighboring files, recent edits, tool results, or model-specific prompt wrappers.

This extension counts the text you are viewing with an OpenAI-compatible tokenizer. It is useful for estimating file and selection size, not for reconstructing Cursor's full prompt.

Literal tokenizer control strings in files, such as <|endoftext|>, are counted as ordinary file text. This keeps documentation, fixtures, and tokenizer-related source files countable instead of treating those strings as special prompt markers.

Very large files and binary-like buffers are skipped automatically to keep the editor responsive. You can raise or lower the automatic file-size limit with localTokenCounter.maxFileCharCount.

When the extension reports text length, it labels that value as UTF-16 code units because that is the unit VS Code and JavaScript use for editor text offsets. Multi-cursor selections are counted range-by-range, so the extension does not add newline tokens between selected ranges.

Features

• Status bar token count for the active file.
• Selection-aware counting when text is selected.
• Command Palette actions:
  • Local Token Counter: Count Active File
  • Local Token Counter: Count Selection
  • Local Token Counter: Choose Counter Source
• Configurable OpenAI-compatible encodings:
  • gpt2
  • r50k_base
  • p50k_base
  • p50k_edit
  • cl100k_base
  • o200k_base
• Fixed model dropdown backed by the pinned local tiktoken model-to-encoding mapping.

The extension activates after editor startup so the status bar count is available without running a command first.

Privacy Summary

Local Token Counter does not send file contents, selections, filenames, workspace paths, telemetry, or usage data to external services. Counting happens locally with the pinned tiktoken package.

The extension does not attempt to describe or control unrelated Cursor or VS Code features. If your editor, sync settings, AI features, or other extensions transmit data, those behaviors are separate from Local Token Counter.

Limitations

• Counts are tokenizer estimates for the text shown in the editor, not full Cursor prompt-size measurements.
• Cursor may include hidden context such as rules, chat history, tool results, surrounding files, or model-specific prompt wrappers.
• Model mode uses the pinned local tiktoken model-to-encoding mapping. If a model is not represented by the dropdown, choose the closest explicit encoding instead.
• Very large files and binary-like buffers are skipped by default for responsiveness.

Development

Install dependencies:

npm install

Compile:

npm run compile

Run tests:

npm test

Check dependencies:

npm run audit

Run the full local release gate:

npm run release:check

Package a VSIX through the checked release path:

npm run package:vsix

Publish through the checked Marketplace path:

npm run publish:marketplace

To run locally, open this folder in Cursor or VS Code and launch Run Extension from the debugger.

Packaging

The extension uses the conventional VS Code packaging flow: vsce includes production dependencies and .vscodeignore excludes local-only files. This project does not use a package.json files allowlist, so .vscodeignore and package inspection are the source of truth for what ships.

The ignore list must cover internal plans, generated packages, env files, source maps, tests, git metadata, and other development artifacts.

Before sharing or publishing a .vsix, inspect the exact package contents:

npm run release:check

The package should contain only the compiled extension, production dependencies included by vsce, and public metadata such as README.md. Review the dependency files in the listing too; production dependency contents can change after dependency updates.

The release gate uses the exact-pinned local @vscode/vsce dev dependency so package listing checks do not resolve an unpinned tool at release time.

Use npm run package:vsix and npm run publish:marketplace instead of invoking vsce package or vsce publish directly. The direct vsce commands run vscode:prepublish, but the project scripts also run the package-file boundary check before packaging or publishing.

The release gate also checks root .vsix artifacts. Delete stale packages before final packaging; only the current ${name}-${version}.vsix filename is allowed in the repository root.

Workspace Trust And Remote Workspaces

Local Token Counter declares support for untrusted workspaces because it only reads editor text for local token counting. It does not execute workspace code, write files, or transmit file content.

The extension also declares virtual workspace support. Non-file URI schemes, including virtual and remote documents, are still skipped by default unless localTokenCounter.countNonFileSchemes is enabled.

In remote setups, Cursor or VS Code chooses the extension host where the extension runs. Local Token Counter does not add its own network transfer of file contents.

Settings

• localTokenCounter.countingMode: encoding or model.
• localTokenCounter.encoding: tokenizer encoding used in encoding mode.
• localTokenCounter.model: fixed model dropdown used in model mode. Model values are resolved locally through the pinned tiktoken package.
• localTokenCounter.countSelections: count selected text instead of the full file when a selection is active.
• localTokenCounter.countUntitled: count untitled editors.
• localTokenCounter.countNonFileSchemes: include virtual and non-file documents.
• localTokenCounter.displayOnRightSide: show the status bar item on the right.
• localTokenCounter.debounceMs: update delay for normal files.
• localTokenCounter.largeFileDebounceMs: update delay for large files.
• localTokenCounter.largeFileCharThreshold: character threshold for large-file debounce.
• localTokenCounter.maxFileCharCount: maximum UTF-16 code units counted automatically for a full file or selection.

Local Token Counter: Choose Counter Source asks whether to save the source in workspace settings or user settings when a workspace is open. The explicit Local Token Counter: Count Selection command always counts the current selection; localTokenCounter.countSelections only controls the automatic status bar behavior.

Dependency Notes

Runtime dependency:

• tiktoken@1.0.22, exact pinned in package.json and package-lock.json.

Why this dependency:

• It supports OpenAI-compatible encodings locally.
• It has zero npm runtime dependencies.
• It is MIT licensed.
• Public package metadata and vulnerability databases checked during planning reported no known direct vulnerabilities for the pinned version.

Tradeoff:

• tiktoken ships a WASM tokenizer. The WASM module is loaded from the installed package by the extension host and does not give the extension any additional file, network, shell, or environment access.
• The extension intentionally imports the package root, not tiktoken/load. tiktoken/load can fetch tokenizer data and must not be introduced without a separate privacy review.
• Any tokenizer dependency update must re-run npm run release:check, inspect npm ls --all, and review the local vsce ls output because production dependency files can change between versions.

Fallback if WASM packaging becomes a problem:

• js-tiktoken@1.0.21, with base64-js@1.5.1 pinned via lockfile. This has a larger JavaScript footprint and one runtime dependency, so it is not the default.

Support

Use GitHub Issues for bug reports and support requests. Do not include private file contents in public issues.

For security issues, use GitHub's private vulnerability reporting / Security Advisories for the repository. Do not disclose suspected vulnerabilities publicly until they have been reviewed.