Tokenlint - AI Token Counter

Real-time LLM token counting and cost estimates for OpenAI, Claude, Gemini & more - The smart way to monitor token usage and costs while developing AI applications. Now with custom model support - add your own models and keep pricing up-to-date!

✨ Features

🔢 Real-time Token Counting

• Instant feedback - See token counts update as you type
• Multiple models - Support for GPT-4, Claude, Gemini, and more
• Visual indicators - Color-coded status bar with usage warnings
• Selection support - Count tokens for selected text vs entire document

💰 Cost Estimation

• Accurate pricing - Up-to-date pricing for all supported models
• Real-time costs - See exact costs as you write
• Cost comparison - Compare costs across different models
• Budget awareness - Visual warnings when approaching limits

🎯 Smart Optimization

• Intelligent suggestions - Get tips to reduce token usage
• Model recommendations - Suggestions for cheaper alternatives
• Usage analysis - Identify repetitive or inefficient content
• Optimization tooltips - Hover for detailed improvement suggestions

🎨 Visual Feedback

• Status bar integration - Clean, informative status display
• Color coding - Green (safe), Yellow (warning), Red (error)
• Inline diagnostics - Squiggly lines when exceeding limits
• Rich tooltips - Detailed information on hover

🔧 Custom Models & Pricing

• Add custom models - Support for any AI model with custom configuration
• Update pricing - Override pricing when models change rates
• Easy management - Commands and UI to manage your custom models
• Settings integration - All configurations stored in VS Code settings
• Flexible providers - Support for OpenAI, Anthropic, Google, and custom providers

🚀 Quick Start

1. Install the extension from the VS Code Marketplace
2. Open any file - Token counting starts automatically
3. Check the status bar - See your token count and cost
4. Click the status bar - Switch between AI models
5. Select text - Get selection-specific token counts
6. Add custom models - Use Cmd+Shift+P → "Add Custom Model" for your own models

🌐 Website & Resources

• 🏠 Official Website: tokenlint.com - Features, demos, and tutorials
• 📦 VS Code Marketplace: Install Extension
• 💻 GitHub Repository: Source Code

📊 Supported Models

Built-in Models

Custom Models

✨ NEW: Add your own models! See Custom Models Guide for details.

• Any AI Provider - OpenAI, Anthropic, Google, or custom APIs
• Custom Pricing - Set your own input/output rates
• Flexible Configuration - Custom token limits and context windows
• Override Pricing - Update existing model pricing when rates change

🎯 Usage Examples

Status Bar Display

✓ 1,234/16,384 (GPT-4o Mini) $0.0002

• ✓ = Safe usage (under 80%)
• ⚠ = Warning (80-99% of limit)
• ❌ = Error (over limit)

Selection Counting

Select any text and use:

• Keyboard: Cmd+Shift+T (Mac) or Ctrl+Shift+T (Windows/Linux)
• Right-click: "Tokenlint: Count Selection"
• Command Palette: "Tokenlint: Count Selection"

Model Switching

• Click status bar - Quick model picker
• Command Palette - "Tokenlint: Select Model"
• Persistent selection - Your choice is remembered

🔧 Commands

💡 Pro Tips

1. Use the cheapest model for your needs - GPT-4o Mini and Claude Haiku are very cost-effective
2. Monitor the status bar - Yellow means you're approaching limits
3. Select before counting - Get precise counts for specific sections
4. Hover for suggestions - Get optimization tips on selected text
5. Switch models easily - Click the status bar to compare costs
6. Add custom models - Include your own fine-tuned or enterprise models
7. Update pricing - Keep costs accurate when model pricing changes
8. Try different providers - Compare costs across OpenAI, Anthropic, and Google

🔧 Configuration

API Keys for Exact Tokenization

By default, Tokenlint uses calibrated approximations for Claude models (~90-95% accuracy). For 100% exact tokenization, you can optionally configure your own API keys:

Anthropic API Key (Optional)

1. Get your API key:

   • Visit Anthropic Console
   • Sign up/log in and create an API key
   • Copy the key (starts with sk-ant-api03-...)

2. Configure in VSCode:

   Option A: Settings UI (Recommended)

   • Open VSCode Settings: Ctrl+, (Windows/Linux) or Cmd+, (Mac)
   • Search for "tokenlint"
   • Find "Anthropic Api Key" and paste your key
   • Optionally set "System Prompt" (e.g., "You are a helpful assistant")

   Option B: Settings JSON

   • Press Ctrl+Shift+P / Cmd+Shift+P
   • Type "Preferences: Open Settings (JSON)"
   • Add:

   {
       "tokenlint.anthropicApiKey": "sk-ant-api03-your-key-here",
       "tokenlint.systemPrompt": "You are a helpful assistant"
   }
   

3. Privacy: Your API key is stored securely in VSCode's encrypted storage

What data is sent to Anthropic:

• Only the text you're analyzing for token counting
• Your optional system prompt
• No other data, files, or personal information

Benefits:

• ✅ Exact token counts for Claude models (100% accuracy)
• ✅ Real-time cost estimates with perfect precision
• ✅ System prompt support for more accurate context

Without API key:

• ✅ Still works perfectly with approximation algorithm
• ✅ No external API calls
• ✅ ~90-95% accuracy for Claude models

Custom Models & Pricing (New!)

Tokenlint now supports adding your own custom AI models and updating pricing for existing models. This is perfect for:

• New models released before the extension is updated
• Enterprise models or fine-tuned models specific to your organization
• Pricing updates when model costs change frequently
• Custom endpoints for local or internal AI services

Quick Setup

Via Command Palette (Recommended):

1. Cmd+Shift+P → "Tokenlint: Add Custom Model"
2. Follow the step-by-step wizard to configure your model
3. Your model appears in the model switcher immediately

Via Settings:

1. Open VS Code Settings (Cmd+,)
2. Search for "tokenlint custom models"
3. Add your model configuration in JSON format

Example Custom Model

{
  "tokenlint.customModels": {
    "my-custom-gpt": {
      "name": "My Custom GPT",
      "maxTokens": 4096,
      "contextWindow": 128000,
      "pricing": {
        "input": 2.0,
        "output": 6.0
      },
      "provider": "openai",
      "tokenizer": "gpt"
    }
  }
}

Pricing Overrides

Keep costs up-to-date when model pricing changes:

{
  "tokenlint.pricingOverrides": {
    "gpt-4o": {
      "input": 4.0,
      "output": 12.0
    }
  }
}

📚 Full Documentation: See Custom Models Guide for complete setup instructions and examples.

Model Selection

📈 Why Use Tokenlint?

• Save money - Avoid expensive API calls with accurate cost estimates
• Prevent errors - Get warnings before hitting token limits
• Optimize prompts - Smart suggestions to reduce token usage
• Compare models - Find the best model for your budget
• Professional workflow - Seamless integration with your development process
• Extensible - Add your own custom models and update pricing as needed
• Always current - Keep up with new models and pricing changes
• Enterprise ready - Support for internal and fine-tuned models

🐛 Issues & Feedback

Found a bug or have a feature request? Please open an issue on GitHub.

📄 License

MIT License - see LICENSE for details.

🤝 Contributing

🌟 About This Project

Tokenlint is an open source project created to provide transparent, accurate token counting for AI development. The code is publicly available for learning, auditing, and building trust in the tokenization accuracy.

📖 Open Source Philosophy

• Transparency: All tokenization algorithms and cost calculations are open for inspection
• Trust: You can verify exactly how your tokens are counted and costs calculated
• Learning: Feel free to study the code and implementation techniques
• Auditing: Security-conscious users can review the API integration code

🤝 Contribution Policy

This project is currently maintained by the core team and not actively seeking external contributions. While the code is open source under the MIT license, we're focused on maintaining a stable, reliable extension rather than community development.

If you find issues:

• 🐛 Bug reports are welcome via GitHub Issues
• 💡 Feature suggestions can be shared, though implementation is at our discretion
• 🔍 Security issues should be reported responsibly

For developers:

• ✅ Feel free to fork for personal use or learning
• ✅ Use the code as reference for your own projects
• ✅ Study the tokenization algorithms and API integrations

Made with ❤️ by the Tokenlint team 🌐 Website • 📦 VS Code Marketplace • 💻 GitHub