Katoid Extension for Visual Studio Code

Visual Studio Code extension for Katoid with support for query execution and results visualization.

Features

• Syntax Highlighting: Full syntax highlighting for Katoid query files (.query.ktd)
• Custom File Icons: Distinctive orange icons for Katoid query files
• Query Execution: Execute queries against Katoid engine via HTTP or CLI
• Dual Engine Support:
  • HTTP Mode: Execute queries via HTTP POST requests with partial results handling and automatic compression detection
  • CLI Mode: Execute queries via command-line engine with stdout/stderr parsing
• Error Display: Comprehensive error handling with inline diagnostics and detailed error messages
• Results Visualization: Interactive results display with:
  • Accordion or combobox view modes
  • Paginated tables with configurable rows per page
  • Channel and summary results display
  • Execution time and response size metrics
• Debug Sections: Collapsible debug panels showing:
  • Query sent to engine (with lazy loading for large content)
  • Response received from engine
  • Download buttons for query and response JSON
  • Size warnings for large content (>1MB)
• Configuration Management: Easy configuration through VS Code settings
• Progress Tracking: Real-time progress indicator in status bar

Installation

There are three ways to install the Katoid extension:

Option 1: Install from VSIX file (Recommended for team distribution)

If you have received a .vsix file:

From VS Code Interface:

1. Open VS Code
2. Go to Extensions view (Ctrl+Shift+X / Cmd+Shift+X)
3. Click the "..." menu at the top of the Extensions view
4. Select "Install from VSIX..."
5. Navigate to the vscode-katoid-0.1.0.vsix file and select it

From Command Line:

code --install-extension vscode-katoid-0.1.0.vsix

Activating File Icons:

After installation, activate the Katoid file icons:

1. Open Command Palette (Ctrl+Shift+P / Cmd+Shift+P)
2. Type "Preferences: File Icon Theme"
3. Select "Katoid File Icons"

Option 2: Install from Source Code

If you have the source code:

1. Clone or copy the repository
2. Install dependencies:

   npm install
   

3. Compile the extension:

   npm run compile
   

4. Copy the entire folder to VS Code extensions directory:
   • macOS/Linux: ~/.vscode/extensions/vscode-katoid-0.1.0/
   • Windows: %USERPROFILE%\.vscode\extensions\vscode-katoid-0.1.0\
5. Restart VS Code

Option 3: Development Mode

For development and testing:

1. Clone the repository
2. Install dependencies:

   npm install
   

3. Compile the extension:

   npm run compile
   

4. Press F5 in VS Code to open a new window with the extension loaded

Creating a VSIX Package

To create a distributable .vsix file:

# Compile the extension
npm run compile

# Create the VSIX package manually
zip -r vscode-katoid-0.1.0.zip . -x "node_modules/*" -x ".git/*" -x "*.vsix" -x "src/*" -x "*.ts" -x "tsconfig.json"
mv vscode-katoid-0.1.0.zip vscode-katoid-0.1.0.vsix

Updating the Extension

To update to a new version:

1. Uninstall the current version from VS Code
2. Install the new .vsix file following the steps above
3. Restart VS Code if necessary

Usage

Creating a Query File

1. Create a new file with .query.ktd extension (or .json with cq_ prefix for legacy compatibility)
2. Write your query core (facets, channels, summaries):

{
  "facets": [
    {
      "name": "by_conversation",
      "group_events_by": [
        { "name": "conversation_id", "expr": ["event_field", "conversation_id"] }
      ],
      "calculators": [
        { "name": "conversation_id", "op": "any", "expr": ["event_field", "conversation_id"] }
      ]
    }
  ],
  "channels": [
    {
      "name": "ch_global",
      "sources": [
        {
          "facet": "by_conversation",
          "group_by": [{ "name": "Global", "expr": "Global" }],
          "calculators": [{ "name": "Total", "op": "count" }]
        }
      ]
    }
  ]
}

Executing Queries

Default Execution:

• Press Ctrl+Shift+E (Mac: Cmd+Shift+E)
• Or right-click in the editor and select "Katoid Query: Execute Query"
• Or use the Command Palette: Katoid Query: Execute Query

Custom Parameters:

• Press Ctrl+Shift+Alt+E (Mac: Cmd+Shift+Alt+E)
• Or use Command Palette: Katoid Query: Execute Query with Custom Parameters...
• Enter custom values for from, to, eval_as_of, and decimation

Uploading Queries to Database

The extension integrates with the update_db tool to upload queries directly to the database.

Prerequisites:

• Each queries folder needs:
  • update_db.sh - Wrapper script that calls update_db.py
  • update_db_config.json - Configuration with datasources and targets
  • $KTD_DEV environment variable must be set

Quick Upload to Local:

• Press Ctrl+K U (Mac: Cmd+K U) - Upload current query to local environment
• No confirmation needed, uploads directly

Upload with Environment Selection:

• Right-click in editor and select "Katoid Query: Upload Query..."
• Or use Command Palette: Katoid Query: Upload Query...
• Select target environment (Local/Staging/Production)
• Confirm if required by environment settings

Upload Entire Folder:

• Right-click on a folder in Explorer
• Select "Katoid Query: Upload All Queries in Folder..."
• Select target environment
• Confirm upload (shows number of queries to upload)

Upload Output:

• Real-time output in "Katoid Upload" channel
• Shows progress, success/failure for each query
• Preserves emojis and formatting from update_db.py script

Viewing Results

After execution, a webview panel opens showing:

Metadata Section:

• Execution Time (ms or seconds)
• Response Size (bytes, KB, or MB)
• Progress percentage
• Cache usage status
• Number of channels/summaries

Results Display:

The extension supports two visualization modes (configurable in settings):

1. Accordion Mode (default): Each channel/summary in expandable sections
2. Combobox Mode: Dropdown selector to switch between results

Each result shows:

• Paginated table with configurable rows per page (default: 100)
• Navigation buttons (First, Previous, Next, Last)
• Export buttons for CSV and JSON formats

Debug Sections:

Collapsible sections showing:

• Query Sent to Engine: The complete query JSON sent to the backend
• Response Received: The full response JSON from the engine

Debug features:

• Lazy loading: Content loads only when you expand the section
• Size warning: Displays alert if content exceeds 1MB before loading
• Download buttons: Save query or response to JSON file
• Formatted JSON display with syntax highlighting

Configuration

Open VS Code Settings (Ctrl+, or Cmd+,) and search for "Katoid":

Engine Configuration

HTTP Mode:

{
  "katoid.engine.mode": "http",
  "katoid.engine.http.url": "http://localhost:8080/query",
  "katoid.engine.http.dataSource": "cbk1_cj"
}

CLI Mode:

{
  "katoid.engine.mode": "cli",
  "katoid.engine.cli.executablePath": "/path/to/katoidd",
  "katoid.schema.path": "/path/to/schema",
  "katoid.engine.cli.kbinsPath": "/path/to/data.kcb",
  "katoid.engine.cli.extraArgs": "--custom_param value"
}

Query Parameters

{
  "katoid.query.from": "2021-11-25",
  "katoid.query.to": "2025-11-23",
  "katoid.query.evalAsOf": "2025-11-25",
  "katoid.query.decimation": 1.0
}

UI Configuration

{
  "katoid.ui.resultsMode": "accordion",           // or "combobox"
  "katoid.ui.maxRowsPerPage": 100,                // pagination size
  "katoid.ui.autoExpandSingleChannel": true       // auto-expand single results
}

Debug Configuration

{
  "katoid.debug.enableLogging": true              // enable detailed logs
}

Multi-Environment Configuration

The extension supports environment-specific configurations using VSCode's folder settings. This allows you to have different configurations for production, staging, development, etc., all in the same workspace.

Configuration Scopes

Settings are organized into two categories:

Environment-Specific Settings (scope: resource) These can be configured per folder and will override global settings:

• katoid.engine.mode - Engine execution mode (http or cli)
• katoid.engine.http.url - HTTP engine URL
• katoid.engine.http.dataSource - Data source identifier
• katoid.schema.path - Katoid schema file path (for engine and tools)
• katoid.engine.cli.kbinsPath - CLI data file path
• katoid.engine.cli.extraArgs - CLI additional arguments
• katoid.query.from - Default from date
• katoid.query.to - Default to date
• katoid.query.evalAsOf - Default eval_as_of timestamp
• katoid.query.decimation - Default decimation value

Global Settings (scope: window or machine) These apply to all environments:

• katoid.engine.cli.executablePath - Path to katoidd executable (same for all envs)
• katoid.ui.resultsMode - Results visualization mode
• katoid.ui.maxRowsPerPage - Pagination size
• katoid.ui.autoExpandSingleChannel - Auto-expand behavior
• katoid.ui.maxResponseSizeMB - Response size limit
• katoid.ui.maxRowsInMemory - Memory limit for results
• katoid.export.jsonFormat - JSON export format
• katoid.export.externalEditor - External editor path
• katoid.debug.enableLogging - Debug logging toggle

Setting Up Multiple Environments

1. Create a Multi-Root Workspace

Create or edit your .code-workspace file:

{
  "folders": [
    {
      "name": "production",
      "path": "queries/production"
    },
    {
      "name": "staging",
      "path": "queries/staging"
    },
    {
      "name": "development",
      "path": "queries/development"
    }
  ],
  "settings": {
    // Global settings here
    "katoid.engine.cli.executablePath": "/usr/local/bin/katoidd",
    "katoid.ui.maxRowsPerPage": 100,
    "katoid.ui.resultsMode": "accordion"
  }
}

2. Configure Each Environment

In each environment folder, create .vscode/settings.json:

queries/production/.vscode/settings.json:

{
  "katoid.engine.mode": "http",
  "katoid.engine.http.url": "https://prod.example.com:8080/query",
  "katoid.engine.http.dataSource": "prod_db",
  "katoid.query.from": "2024-01-01",
  "katoid.query.to": "2024-12-31",
  "katoid.query.evalAsOf": "2024-12-31"
}

queries/staging/.vscode/settings.json:

{
  "katoid.engine.mode": "http",
  "katoid.engine.http.url": "https://staging.example.com:8080/query",
  "katoid.engine.http.dataSource": "staging_db",
  "katoid.query.from": "2024-06-01",
  "katoid.query.to": "2024-12-31"
}

queries/development/.vscode/settings.json:

{
  "katoid.engine.mode": "cli",
  "katoid.schema.path": "/data/dev/schema.json",
  "katoid.engine.cli.kbinsPath": "/data/dev/data.kcb",
  "katoid.query.from": "2024-11-01",
  "katoid.query.to": "2024-11-30"
}

3. Example Project Structure

my-project/
├── my-project.code-workspace
├── queries/
│   ├── production/
│   │   ├── .vscode/
│   │   │   └── settings.json          # Production config
│   │   ├── query1.kql
│   │   ├── query2.kql
│   │   └── reports/
│   │       └── monthly.kql
│   │
│   ├── staging/
│   │   ├── .vscode/
│   │   │   └── settings.json          # Staging config
│   │   ├── test-query.kql
│   │   └── experiments/
│   │       └── new-feature.kql
│   │
│   └── development/
│       ├── .vscode/
│       │   └── settings.json          # Development config
│       ├── debug-query.kql
│       └── sandbox/
│           └── scratch.kql

4. How It Works

• When you open a .query.ktd file in queries/production/, the extension automatically uses production settings
• When you open a .query.ktd file in queries/staging/, it uses staging settings
• Subfolder structure is fully supported - queries/production/reports/monthly.query.ktd inherits production settings
• The global cli.executablePath is shared across all environments
• UI preferences (pagination, display mode) are consistent across all environments

Important Notes

⚠️ VSCode Workspace Folder Requirement: Environment-specific settings only work when the folder is a workspace folder in a multi-root workspace. Settings in subdirectories that are not workspace folders will be ignored.

Correct ✅:

{
  "folders": [
    { "name": "prod", "path": "queries/production" },     // Workspace folder
    { "name": "staging", "path": "queries/staging" }      // Workspace folder
  ]
}

Incorrect ❌:

{
  "folders": [
    { "name": "queries", "path": "queries" }              // Parent folder only
  ]
}
// Settings in queries/production/.vscode/settings.json will NOT work

Verification

To verify your environment configuration is working:

1. Enable debug logging: "katoid.debug.enableLogging": true
2. Open a query file in your environment folder
3. Execute the query
4. Check View → Output → Katoid for the configuration being used
5. You should see the environment-specific values in the execution log

Architecture

Project Structure

vscode-katoid/
├── src/
│   ├── extension.ts              # Extension entry point and command handlers
│   ├── engine/
│   │   ├── engineClient.ts       # Abstract engine interface
│   │   ├── httpEngineClient.ts   # HTTP implementation with compression detection
│   │   └── cliEngineClient.ts    # CLI implementation
│   ├── query/
│   │   ├── queryTypes.ts         # TypeScript type definitions
│   │   ├── queryBuilder.ts       # Query construction logic
│   │   └── queryExecutor.ts      # Query execution orchestrator
│   ├── ui/
│   │   ├── resultsWebview.ts     # Results visualization webview
│   │   ├── errorHandler.ts       # Error display and diagnostics
│   │   └── statusBar.ts          # Status bar progress indicator
│   ├── errors/
│   │   └── EngineError.ts        # Custom error class
│   └── utils/
│       ├── configuration.ts      # Configuration management
│       └── logger.ts             # Logging utility
├── icons/
│   ├── query-file-icon.svg      # Query file icon
│   ├── schema-file-icon.svg     # Schema file icon
│   ├── katoid-icon.svg          # Sidebar icon
│   └── katoid-icon-theme.json   # Icon theme definition
├── syntaxes/
│   ├── ktd-query.tmLanguage.json    # Query syntax grammar
│   └── ktd-schema.tmLanguage.json   # Schema syntax grammar
├── examples/
│   └── sample.query.ktd         # Example query
└── package.json                 # Extension manifest

How It Works

1. Query Core: You edit only the "core" of the query (facets, channels, summaries)
2. Query Builder: The extension wraps your core with execution parameters (from, to, eval_as_of, etc.)
3. Engine Client: Executes the complete query via HTTP or CLI
   • HTTP: Automatically detects and decompresses gzip/deflate responses (even without proper headers)
   • CLI: Captures stdout/stderr and parses JSON output
4. Progress Handling: Automatically follows partial results until progress = 1.0
   • Updates status bar in real-time
   • Tracks execution time from start to completion
5. Error Detection: Catches and displays errors from HTTP responses or stderr
   • Shows inline diagnostics with line/column information
   • Logs detailed error information to output channel
6. Results Display: Shows results in interactive webview with:
   • Execution metrics (time, size, progress)
   • Paginated tables for channels and summaries
   • Debug sections with lazy loading
   • Export functionality for CSV/JSON

Error Handling

The extension handles errors from the engine in multiple ways:

• HTTP Mode: Detects errors in HTTP status codes and response JSON
• CLI Mode: Captures errors from stderr and process exit codes
• Display: Shows errors inline with diagnostic markers and in the Output Channel

Example error response:

{
  "error": {
    "type": "query_parse_error",
    "message": "Invalid field reference",
    "line": 15,
    "column": 25,
    "details": "Field 'non_existent_field' does not exist"
  }
}

Commands

All commands are available via the Command Palette (Ctrl+Shift+P / Cmd+Shift+P):

• Katoid Query: Execute Query - Execute the current query with default parameters

  • Keyboard shortcut: Ctrl+Shift+E (Cmd+Shift+E on Mac)
  • Also available in editor context menu and title bar

• Katoid Query: Execute Query with Custom Parameters... - Execute with custom from/to/evalAsOf/decimation values

  • Keyboard shortcut: Ctrl+Shift+Alt+E (Cmd+Shift+Alt+E on Mac)
  • Opens input dialogs for each parameter

• Katoid: Show Configuration - Open Katoid settings in VS Code settings UI

• Katoid Query: Cancel Running Query - Cancel a query that is currently executing

  • Available in editor title bar when a query is running

• Katoid Query: Export Results to CSV/JSON - Export query results

  • Available after query execution completes
  • Use export buttons in the results panel

• Katoid Query: Copy Error to Clipboard - Copy error details to clipboard

  • Available when an error occurs

• Katoid Query: Go to Error Location - Navigate to error location in the query

  • Available when an error with line/column information occurs

Development Status

✅ Completed

• Project structure and configuration
• TypeScript type definitions
• Syntax highlighting for Katoid query and schema files
• Custom file icons with theme support
• Query builder (core → complete query)
• HTTP engine client with:
  • Automatic gzip/deflate compression detection via magic bytes
  • Partial results handling with progress tracking
  • Custom headers support (Katoid-Progress, Katoid-Req-Id)
• CLI engine client with error handling
• Query executor with execution time tracking
• Results visualization webview with:
  • Accordion and combobox view modes
  • Paginated tables
  • Channel and summary display
  • Execution metrics (time, size, progress, cache usage)
• Debug sections with:
  • Lazy loading for large content
  • Download functionality for query and response
  • Size warnings (>1MB threshold)
• Error display system:
  • Inline diagnostics with line/column highlighting
  • Error details in output channel
  • Copy and navigation commands
• Status bar with real-time progress indicator
• Configuration management
• Logging utilities
• Query cancellation support

📋 Planned

• Auto-completion for Katoid query keywords
• Query validation and linting
• Snippet library for common patterns
• Query history and favorites
• Performance profiling tools
• Schema file support with validation

Troubleshooting

File Icons Not Showing

If Katoid files don't show the custom icons:

1. Open Command Palette (Ctrl+Shift+P / Cmd+Shift+P)
2. Type "Preferences: File Icon Theme"
3. Select "Katoid File Icons"
4. Restart VS Code if necessary

Query Execution Fails

Check Engine Configuration:

• Verify katoid.engine.mode is set correctly ("http" or "cli")
• For HTTP mode: Check katoid.engine.http.url is reachable
• For CLI mode: Check katoid.engine.cli.executablePath points to valid executable

Check Output Channel:

• Open Output panel: View → Output
• Select "Katoid" from the dropdown
• Look for detailed error messages and logs

Enable Debug Logging:

1. Set katoid.debug.enableLogging to true in settings
2. Re-run the query
3. Check the Katoid output channel for detailed logs

Compressed Response Errors

If you see "Invalid JSON response" with binary characters:

• The extension automatically detects and decompresses gzip/deflate responses
• If the error persists, check the Katoid output channel for compression detection logs
• The engine may be sending a different compression format

Large Response Performance

For queries returning large results:

• Increase katoid.ui.maxRowsPerPage for better pagination
• Use export functionality instead of viewing in browser
• Consider adding decimation to reduce result size

Extension Not Activating

If the extension doesn't load:

1. Check VS Code version (requires ^1.80.0 or later)
2. Reinstall the extension
3. Check VS Code's Developer Tools (Help → Toggle Developer Tools) for errors
4. Try reloading the window (Developer: Reload Window)

Contributing

This extension is currently in active development. Contributions are welcome!

Technical Details

Engine Communication

HTTP Protocol:

The extension uses a progressive query execution model:

1. POST /query with CompleteQuery
   → Response: { reqId: "abc", progress: 0.3, ... }
   → Headers: Katoid-Progress: 0.3, Katoid-Req-Id: abc

2. POST /query with "abc" (reqId from previous response)
   → Response: { reqId: "abc", progress: 0.7, ... }
   → Headers: Katoid-Progress: 0.7, Katoid-Req-Id: abc

3. POST /query with "abc"
   → Response: { progress: 1.0, selection_results: [...] }
   → Headers: Katoid-Progress: 1.0

Automatic Compression Detection:

The HTTP client automatically detects and decompresses responses:

• Checks content-encoding header for gzip/deflate
• Falls back to magic byte detection if header is missing:
  • gzip: bytes start with 0x1f 0x8b
  • deflate: bytes start with 0x78
• This ensures compatibility even when servers don't set proper headers

CLI Protocol:

katoidd --tq_query query.json \
        --tq_schema schema_path \
        --tq_kbins data.kcb \
        --testquery_print \
        --logflush 1

Output via stdout as JSON, errors via stderr.

Complete Query Structure

The extension transforms your query core:

{
  "facets": [...],
  "channels": [...]
}

Into a complete query:

{
  "data_source": "cbk1_cj",
  "from": "2021-11-25",
  "to": "2025-11-23",
  "eval_as_of": "2025-11-25",
  "decimation": 1,
  "aggregates": {
    "ids_for_aggregates": false,
    "facets": [...],
    "channels": [...]
  },
  "selections": [{ "name": "all" }]
}

Version History

0.2.0 (Current)

Breaking Changes:

• File extension changed from .kql to .query.ktd
• Language ID changed from kql to ktdQuery
• Command namespace changed from kql.* to katoid.*
• Settings namespace reorganized to katoid.* with hierarchical structure

Features:

• Extensible type system supporting multiple Katoid file types (Query, Schema, etc.)
• New .query.ktd file extension for query files
• New .schema.ktd file extension for schema files (prepared for future)
• Backward compatibility with legacy cq_*.json files
• Updated file icons for different types
• Reorganized settings namespace for better structure

0.1.0

Features:

• Initial release with full query execution support
• Syntax highlighting for query files
• Custom file icons
• HTTP and CLI engine modes
• Automatic compression detection (gzip/deflate)
• Progressive query execution with real-time progress
• Interactive results visualization (accordion/combobox modes)
• Paginated table display with export functionality
• Debug sections with lazy loading and download buttons
• Execution time and response size tracking
• Comprehensive error handling with diagnostics
• Status bar integration
• Query cancellation support

Configuration:

• Engine mode selection (HTTP/CLI)
• Query parameters (from, to, eval_as_of, decimation)
• UI customization (results mode, pagination, auto-expand)
• Debug logging toggle

License

MIT

Support

Output Channel: For debugging and logs, check the "Katoid" output channel in VS Code:

• View → Output
• Select "Katoid" from dropdown

Configuration Issues: All settings are available under "Katoid" in VS Code settings (Ctrl+, / Cmd+,)

File Issues: For problems with the extension installation or updates, try:

1. Uninstall the extension
2. Reload VS Code window
3. Reinstall from the .vsix file