JSON to TOON Transformer

A VS Code/Cursor extension that transforms JSON into TOON (Token-Oriented Object Notation) format with convenient keyboard shortcuts. Uses the official @toon-format/toon library.

Features

• Quick Copy: Transform and copy JSON as TOON format with Option+Command+C (Mac) or Alt+Ctrl+C (Windows/Linux)
• Token Counter: See real-time token savings for informational purposes (e.g., "Token savings: 45% (150 → 82)") 🎯
• Quick Paste: Paste TOON format back as formatted JSON with Option+Command+V (Mac) or Alt+Ctrl+V (Windows/Linux)
• JSON Validation: Automatically validates JSON before transformation
• Error Notifications: Clear error messages when copying invalid content (plain text, incomplete JSON, etc.)
• Smart Selection: Works with selected text or entire document if nothing is selected
• Official Implementation: Uses the official @toon-format/toon library
• GPT-4 Tokenizer: Accurate token counting using OpenAI's tiktoken library

What is TOON Format?

TOON (Token-Oriented Object Notation) is a data format designed specifically for LLM prompts. It reduces token usage by 30-60% compared to JSON while maintaining readability and structure.

Key Features

• 📊 Tabular format for arrays of uniform objects
• 🔢 Explicit array lengths with [N] notation
• 🏷️ Field headers like {field1,field2,field3}
• ✂️ Minimal syntax - no unnecessary quotes or brackets
• 📏 2-space indentation for nested structures
• 🤖 LLM-optimized for token efficiency

Example Comparison

JSON (156 tokens):

{
  "users": [
    {
      "id": 1,
      "name": "Alice Johnson",
      "role": "admin",
      "active": true
    },
    {
      "id": 2,
      "name": "Bob Smith",
      "role": "user",
      "active": true
    },
    {
      "id": 3,
      "name": "Charlie Davis",
      "role": "moderator",
      "active": false
    }
  ]
}

TOON (89 tokens - 43% reduction):

users[3]{id,name,role,active}:
  1,Alice Johnson,admin,true
  2,Bob Smith,user,true
  3,Charlie Davis,moderator,false

Format Specifications

Objects

key1: value1
key2: value2
nested:
  innerKey: value

Primitive Arrays

tags[3]: javascript,python,rust
numbers[4]: 1,2,3,4

Tabular Arrays (uniform objects)

items[2]{id,name,price}:
  1,Widget,9.99
  2,Gadget,14.50

Mixed/Non-uniform Arrays (list format)

mixed[3]:
  - 42
  - text: Hello
  - [2]: a,b

For more details, see the official TOON specification.

Installation

From Source

1. Clone this repository:

   git clone <repository-url>
   cd ToTOON
   

2. Install dependencies:

   npm install
   

3. Compile the extension:

   npm run compile
   

4. Open the project in VS Code and press F5 to launch the extension in a new Extension Development Host window

From VSIX Package

1. Package the extension:

   npm install -g @vscode/vsce
   vsce package
   

2. Install the generated .vsix file:

   • Open VS Code
   • Go to Extensions view (Cmd+Shift+X or Ctrl+Shift+X)
   • Click the ... menu → "Install from VSIX..."
   • Select the generated .vsix file

Usage

Copy JSON as TOON

1. Select JSON text in your editor (or have cursor in a document with JSON)
2. Press Option+Command+C (Mac) or Alt+Ctrl+C (Windows/Linux)
3. The JSON is always transformed to TOON format and copied to clipboard
4. A notification shows token comparison:
   • Saves tokens: "✅ JSON copied as TOON! 🎯 Token savings: 45.3% (150 → 82)"
   • Same tokens: "✅ JSON copied as TOON! 📊 Same tokens: 100 = 100 (0% difference)"
   • Uses more tokens: "✅ JSON copied as TOON! ⚠️ Token cost: +15.2% (200 → 230)"
5. You decide whether to use the TOON version based on your needs

Example:

• Select the JSON in examples/sample.json
• Press Option+Command+C
• See the TOON format in clipboard (compare with examples/sample.toon)

Paste TOON as JSON

1. Have TOON format content in your clipboard
2. Place cursor where you want to paste
3. Press Option+Command+V (Mac) or Alt+Ctrl+V (Windows/Linux)
4. The TOON format will be converted back to formatted JSON

Example:

• Copy the content from examples/tabular-example.toon
• Press Option+Command+V
• See it converted to JSON format

Error Handling

The extension will show error messages for:

• ❌ Empty content
• ❌ Plain text (not JSON)
• ❌ Malformed/incomplete JSON
• ❌ Invalid TOON format on paste

Keyboard Shortcuts

You can customize these shortcuts in VS Code settings:

• Open Command Palette (Cmd+Shift+P or Ctrl+Shift+P)
• Type "Preferences: Open Keyboard Shortcuts"
• Search for "JSON to TOON"

Why Use TOON?

For LLM Prompts

When working with large datasets in LLM prompts, TOON can significantly reduce token usage:

• Tabular data: 40-60% fewer tokens than JSON
• Nested objects: 20-40% fewer tokens (depending on depth)
• Self-documenting: Array lengths and field headers help LLMs understand structure

Format Advantages

• ✅ Human-readable: Even more readable than JSON
• ✅ Structured: Maintains all JSON data types and nesting
• ✅ Validated: Explicit array lengths catch errors
• ✅ Standard: Based on official specification with multiple implementations

When TOON Typically Saves Tokens

• 📊 Uniform arrays of objects (same fields, primitive values)
• 📈 Large datasets with consistent structure
• 🤖 LLM prompts where token count matters
• 📝 Human-readable data interchange
• 📄 Flat to moderately nested data (2-3 levels)

When TOON May Not Help

• 🔀 Non-uniform data structures
• 🌳 Deeply nested, complex hierarchies (4+ levels)
• 🔌 API responses (where JSON is standard)
• 📦 Long-term storage (wider tool support)

Note: The extension always converts when you request it. Use the token counter to see the impact and decide if TOON is right for your specific use case.

Examples

Simple Object

{"name": "Alice", "age": 30, "active": true}

↓

name: Alice
age: 30
active: true

Nested Object

{
  "user": {
    "id": 1,
    "profile": {
      "name": "Alice"
    }
  }
}

↓

user:
  id: 1
  profile:
    name: Alice

Tabular Data (The Sweet Spot!)

{
  "products": [
    {"sku": "A1", "name": "Widget", "price": 9.99},
    {"sku": "B2", "name": "Gadget", "price": 14.50}
  ]
}

↓

products[2]{sku,name,price}:
  A1,Widget,9.99
  B2,Gadget,14.5

Development

Project Structure

ToTOON/
├── src/
│   ├── extension.ts          # Main extension logic
│   └── toonTransformer.ts    # TOON transformation (uses official lib)
├── examples/
│   ├── sample.json           # Example JSON file
│   ├── sample.toon           # Example TOON (nested objects)
│   ├── tabular-example.json  # Tabular data example
│   └── tabular-example.toon  # Shows TOON's efficiency
├── out/                       # Compiled JavaScript (generated)
├── package.json               # Extension manifest
└── README.md                  # This file

Available Scripts

• npm run compile - Compile TypeScript to JavaScript
• npm run watch - Watch mode for development
• npm run lint - Run ESLint

Building from Source

# Install dependencies
npm install

# Compile TypeScript
npm run compile

# Package extension
vsce package

Requirements

• VS Code version 1.74.0 or higher
• Node.js and npm for development

Resources

• Official TOON Repository
• TOON Format Specification
• TOON Website
• DEV Article: TOON Benchmarks
• When to Use TOON Guide (important!)
• VS Code Extension API

Contributing

Contributions are welcome! This extension uses the official TOON library, so format improvements happen upstream.

For extension-specific improvements:

• Bug fixes
• UX enhancements
• Documentation improvements
• Additional features

License

MIT License - feel free to use this extension in your projects.

Acknowledgments

• Built with @toon-format/toon - the official TOON implementation
• TOON format created by Johann Schopplich and contributors

Release Notes

0.0.2

• ✅ Correct TOON implementation using official library
• 🎯 NEW: Real-time token counter with savings percentage
• 📊 Uses GPT-4 tokenizer (tiktoken) for accurate counts
• 📚 Comprehensive documentation about when to use TOON
• 🔗 References to official TOON benchmarks

0.0.1 (deprecated)

• ❌ Incorrect custom implementation (not real TOON format)
• Basic copy/paste functionality

Transform your JSON for LLM efficiency! 🚀