Tokemizer VS Code Extension

Tokemizer is a VS Code extension that can route prompts through Tokemizer first, then forward the optimized request to a selected language model inside VS Code through the official VS Code Language Model API.

Features

• Chat participant: use @tokemizer to optimize prompts before they reach the target model.
• Language-model provider: select Tokemizer directly as a chat model so Tokemizer can optimize the request and forward it to your chosen target model.
• Image attachments: attach images when using the Tokemizer provider and the extension will forward them to the selected target model when it supports image input.
• Prompt insights: view token savings, telemetry, and request history in Tokemizer: Open Insights.
• Target model selection: choose the downstream model Tokemizer should forward to with Tokemizer: Set Target Model.
• Safe fallback handling: if the optimization endpoint fails, the extension surfaces the error and continues with the original prompt.

Flow

1. Open VS Code Chat and send a message to @tokemizer, or select Tokemizer as the chat model.
2. The extension sends the prompt to Tokemizer using the API key saved by Tokemizer: Set API Key.
3. Tokemizer returns an optimized prompt and telemetry.
4. The extension forwards the optimized prompt to the configured target model through vscode.lm.
5. The streamed response appears in VS Code Chat.
6. Token savings and telemetry are available in Tokemizer: Open Insights.

Chat Participant

Use @tokemizer in VS Code Chat to route a prompt through Tokemizer before it reaches the selected VS Code chat model.

Examples:

@tokemizer Explain this code and point out risks.
@tokemizer Refactor this function for readability.
@tokemizer Summarize this long prompt and preserve all requirements.

For chat requests, the extension sends the current @tokemizer prompt plus available Tokemizer chat history and VS Code chat references to Tokemizer for optimization.

Language Model Provider

Tokemizer also registers a selectable language-model provider named Tokemizer. When you choose it in the model picker, the extension captures the request, sends it to Tokemizer for optimization, and then forwards the optimized request to the target model you selected with Tokemizer: Set Target Model.

The provider path preserves tool options and tool-call response parts so VS Code can continue the public tool-calling flow when the host supports it.

If you attach images, the provider keeps those attachments in the forwarded request. The downstream target model must also support image input.

Commands

• Tokemizer: Open Insights
• Tokemizer: Set API Key
• Tokemizer: Set Optimization Mode
• Tokemizer: Set Target Model

Settings

• tokemizer.optimizeEndpoint
• tokemizer.optimizationMode
• tokemizer.debugLogging

The default optimize endpoint is:

https://www.tokemizer.com/api/v1/optimize

For local backend testing, set tokemizer.optimizeEndpoint to:

http://localhost:8000/api/v1/optimize

tokemizer.optimizationMode controls the compression preset sent as optimization_mode:

• conservative: Lite
• balanced: Standard, and the default
• maximum: Aggressive

Seamless Model Proxy

Run Tokemizer: Set Target Model to choose the real VS Code language model that Tokemizer forwards requests to.

When Tokemizer is selected as the chat model, the extension optimizes the latest user text prompt, forwards the request to the configured target model, and streams response parts back to VS Code.

Security

• This extension does not access Copilot, Codex, or provider tokens.
• This extension stores the Tokemizer API key in VS Code SecretStorage.
• This extension does not store OpenAI, Anthropic, OpenRouter, Copilot, or other provider keys.
• This extension does not read Copilot extension storage.
• This extension does not call private Copilot, Codex, or provider APIs.
• This extension only sends the optimized prompt through the official VS Code Language Model API.

Running Locally

1. Open this folder in VS Code:

   tokemizer/extensions/tokemizer-lm-api-spike
   

2. Install dependencies:

   npm install
   

3. Compile:

   npm run compile
   

4. Press F5 in VS Code to launch an Extension Development Host.

5. Run Tokemizer: Set API Key from the Command Palette.

6. Optional: configure tokemizer.optimizeEndpoint for a local backend.

7. Open VS Code Chat and send a message to @tokemizer.