Tokemizer VS Code Extension

This extension optimizes prompts with Tokemizer before sending them to a language model already available inside VS Code through the official VS Code Language Model API.

Flow

1. Open VS Code Chat and send a message to @tokemizer.
2. The extension sends the prompt to Tokemizer using the API key saved by Tokemizer: Set API Key.
3. Tokemizer returns an optimized prompt and telemetry.
4. The extension sends the optimized prompt to the selected VS Code language model using vscode.lm.
5. The streamed response appears in VS Code Chat for @tokemizer.
6. Token savings and telemetry are available in Tokemizer: Open Insights.

VS Code Chat

Use @tokemizer in VS Code Chat to route a prompt through Tokemizer before it reaches the selected VS Code chat model.

Examples:

@tokemizer Explain this code and point out risks.
@tokemizer Refactor this function for readability.
@tokemizer Summarize this long prompt and preserve all requirements.

For chat requests, the extension sends the current @tokemizer prompt plus available Tokemizer chat history and VS Code chat references to Tokemizer for optimization.

Commands

• Tokemizer: Open Insights
• Tokemizer: Set API Key
• Tokemizer: Set Optimization Mode

Settings

• tokemizer.optimizeEndpoint
• tokemizer.optimizationMode
• tokemizer.debugLogging

The default optimize endpoint is:

https://www.tokemizer.com/api/v1/optimize

For local backend testing, set tokemizer.optimizeEndpoint to:

http://localhost:8000/api/v1/optimize

tokemizer.optimizationMode controls the compression preset sent as optimization_mode:

• conservative: Lite
• balanced: Standard, and the default
• maximum: Aggressive

Security

• This extension does not access Copilot, Codex, or provider tokens.
• This extension stores the Tokemizer API key in VS Code SecretStorage.
• This extension does not store OpenAI, Anthropic, OpenRouter, Copilot, or other provider keys.
• This extension does not read Copilot extension storage.
• This extension does not call private Copilot, Codex, or provider APIs.
• This extension only sends the optimized prompt through the official VS Code Language Model API.

Running Locally

1. Open this folder in VS Code:

   tokemizer/extensions/tokemizer-lm-api-spike
   

2. Install dependencies:

   npm install
   

3. Compile:

   npm run compile
   

4. Press F5 in VS Code to launch an Extension Development Host.

5. Run Tokemizer: Set API Key from the Command Palette.

6. Optional: configure tokemizer.optimizeEndpoint for a local backend.

7. Open VS Code Chat and send a message to @tokemizer.