Glassbox AI Testing Extension

🚀 Features

🧪 Intelligent Test Generation

• AI-Powered Test Creation: Generate comprehensive tests for AI prompts automatically
• Multi-Model Support: Works with OpenAI, Anthropic, Google AI, and local models
• Smart Test Types: Unit, integration, performance, and security tests
• Cost-Aware Generation: Built-in cost estimation and management

🔍 Real-Time Prompt Detection

• Automatic Detection: Identify AI prompts in your code instantly
• Multi-Language Support: JavaScript, TypeScript, Python, Java, C#, Go, Rust, PHP, Ruby
• Security Scanning: Detect hardcoded API keys and security issues
• Performance Analysis: Identify expensive API calls and optimization opportunities

💰 Cost Management

• Real-Time Cost Tracking: Monitor API usage and costs
• Threshold Alerts: Set warning and error thresholds
• Budget Controls: Per-test and per-session cost limits
• Currency Support: USD, EUR, GBP

🛠️ Comprehensive Testing

• CLI Integration: Seamless integration with Glassbox CLI
• Multiple Output Formats: JSON, YAML, XML, HTML reports
• Test Organization: Structured test directories and naming
• Validation: Schema validation and test structure verification

📊 Visual Feedback

• Status Bar Integration: Real-time status and cost information
• Code Lens: Inline test generation and execution buttons
• Diagnostics: Hover information and detailed error messages
• Test Results Panel: Comprehensive test result visualization

📸 Screenshots

Command Palette Integration

Test Generation Interface

Cost Management Dashboard

Prompt Detection

Settings Configuration

🎯 Quick Start

1. Installation

1. Install the Extension

   # From VS Code Extensions Marketplace
   # Search for "Glassbox AI Testing"
   

2. Install Glassbox CLI

   # macOS
   brew install glassbox-ai/glassbox/glassbox
   
   # Linux
   curl -fsSL https://glassbox.ai/install.sh | sh
   
   # Windows
   # Download from https://glassbox.ai/download
   

3. Configure Settings

   {
     "glassbox-ai.cliPath": "/usr/local/bin/glassbox",
     "glassbox-ai.aiModels.preferred": "gpt-4",
     "glassbox-ai.costThresholds.warning": 0.10
   }
   

2. Basic Usage

Generate Tests for AI Prompts

1. Open a file containing AI prompts
2. Press Ctrl+Shift+G (or Cmd+Shift+G on Mac)
3. Select the prompt text
4. Choose test generation options

Run Tests

1. Open a test file or AI prompt file
2. Press Ctrl+Shift+R (or Cmd+Shift+R on Mac)
3. View results in the Test Results panel

Detect AI Prompts

1. Open any supported code file
2. Right-click and select "Glassbox: Detect AI Prompts"
3. View detected prompts in the Problems panel

3. Advanced Features

Automatic Test Generation

{
  "glassbox-ai.automaticTestGeneration": {
    "enabled": true,
    "onFileSave": true,
    "includeUnitTests": true,
    "includeSecurityTests": true
  }
}

Local AI Models

{
  "glassbox-ai.aiModels": {
    "preferred": "llama2",
    "local": {
      "enabled": true,
      "endpoint": "http://localhost:11434",
      "model": "llama2:13b",
      "provider": "ollama"
    }
  }
}

📋 Command Reference

Primary Commands (with shortcuts)

File-Specific Commands

Tool Commands

Analysis Commands

⚙️ Configuration

The Glassbox AI extension provides extensive configuration options organized into logical categories.

🗂️ Test Directories

Configure where different types of tests are stored:

{
  "glassbox-ai.testDirectories": {
    "default": ".glassbox/tests",
    "unit": ".glassbox/tests/unit",
    "integration": ".glassbox/tests/integration",
    "e2e": ".glassbox/tests/e2e",
    "performance": ".glassbox/tests/performance",
    "security": ".glassbox/tests/security"
  }
}

🤖 AI Models

Configure AI models for test generation and execution:

{
  "glassbox-ai.aiModels": {
    "preferred": "gpt-4",
    "fallback": "gpt-3.5-turbo",
    "local": {
      "enabled": false,
      "endpoint": "http://localhost:11434",
      "model": "llama2",
      "provider": "ollama"
    },
    "remote": {
      "openai": {
        "enabled": true,
        "models": ["gpt-4", "gpt-3.5-turbo", "gpt-4-turbo"],
        "default": "gpt-4"
      },
      "anthropic": {
        "enabled": true,
        "models": ["claude-3-opus", "claude-3-sonnet", "claude-3-haiku"],
        "default": "claude-3-sonnet"
      },
      "google": {
        "enabled": false,
        "models": ["gemini-pro", "gemini-pro-vision"],
        "default": "gemini-pro"
      }
    }
  }
}

💰 Cost Thresholds

Configure cost management and alerts:

{
  "glassbox-ai.costThresholds": {
    "warning": 0.10,
    "error": 1.00,
    "maxPerTest": 0.05,
    "maxPerSession": 5.00,
    "currency": "USD"
  }
}

⏱️ Timeouts

Configure timeouts for different operations:

{
  "glassbox-ai.timeouts": {
    "testExecution": 30000,
    "testGeneration": 60000,
    "promptDetection": 5000,
    "cliCommand": 30000,
    "modelResponse": 45000,
    "fileOperation": 10000
  }
}

🔄 Automatic Test Generation

Configure automatic test generation behavior:

{
  "glassbox-ai.automaticTestGeneration": {
    "enabled": false,
    "onFileOpen": false,
    "onFileSave": false,
    "onPromptDetection": false,
    "includeUnitTests": true,
    "includeIntegrationTests": false,
    "includePerformanceTests": false,
    "includeSecurityTests": true,
    "maxTestsPerFile": 10,
    "skipExistingTests": true
  }
}

📄 Output Formats

Configure output formats and verbosity:

{
  "glassbox-ai.outputFormats": {
    "testResults": "json",
    "testGeneration": "yaml",
    "diagnostics": "markdown",
    "logs": "text",
    "reports": "html",
    "verbosity": "normal"
  }
}

✅ Validation

Configure validation settings:

{
  "glassbox-ai.validation": {
    "enableSchemaValidation": true,
    "strictMode": false,
    "allowUnknownProperties": true,
    "validateTestStructure": true,
    "validatePromptFormat": true,
    "validateCostEstimates": true
  }
}

🔔 Notifications

Configure notification settings:

{
  "glassbox-ai.notifications": {
    "showSuccess": true,
    "showWarnings": true,
    "showErrors": true,
    "showProgress": true,
    "showCostAlerts": true,
    "showTimeoutAlerts": true,
    "notificationLevel": "normal"
  }
}

🎯 Usage Examples

Example 1: Basic Test Generation

Input File: prompts/chatbot.md

# Customer Service Chatbot

## System Prompt
You are a helpful customer service representative for TechCorp.

## User Message
"I can't log into my account"

## Expected Response
- Acknowledge the issue
- Ask for account details
- Provide troubleshooting steps
- Offer to escalate if needed

Generated Test:

test_name: "Customer Service Login Issue"
description: "Test customer service response to login problems"

input:
  system_prompt: "You are a helpful customer service representative for TechCorp."
  user_message: "I can't log into my account"

validation:
  - type: "response_contains"
    expected: ["acknowledge", "account", "troubleshoot"]
  
  - type: "response_format"
    expected: "professional and helpful tone"
  
  - type: "safety"
    expected: "no personal information requests"

cost_estimate:
  model: "gpt-4"
  estimated_tokens: 150
  estimated_cost: "$0.006"

Example 2: Code-Based Prompt Testing

Input File: src/ai-service.js

const openai = require('openai');

async function generateResponse(prompt) {
  const completion = await openai.chat.completions.create({
    model: "gpt-4",
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      { role: "user", content: prompt }
    ],
    max_tokens: 1000
  });
  
  return completion.choices[0].message.content;
}

Detected Prompts:

• System prompt: "You are a helpful assistant."
• Model: gpt-4
• Max tokens: 1000

Generated Tests:

test_name: "AI Service Response Generation"
description: "Test AI service response generation"

input:
  prompt: "Hello, how can you help me?"

validation:
  - type: "response_quality"
    expected: "helpful and relevant response"
  
  - type: "token_limit"
    expected: "response within 1000 tokens"
  
  - type: "safety"
    expected: "no harmful content"

cost_estimate:
  model: "gpt-4"
  estimated_tokens: 200
  estimated_cost: "$0.008"

Example 3: Local Model Configuration

{
  "glassbox-ai.aiModels": {
    "preferred": "llama2",
    "local": {
      "enabled": true,
      "endpoint": "http://localhost:11434",
      "model": "llama2:13b",
      "provider": "ollama"
    },
    "remote": {
      "openai": { "enabled": false },
      "anthropic": { "enabled": false }
    }
  },
  "glassbox-ai.timeouts.modelResponse": 90000,
  "glassbox-ai.costThresholds": {
    "warning": 0.01,
    "error": 0.10,
    "maxPerTest": 0.02,
    "maxPerSession": 1.00
  }
}

🛠️ Development

Prerequisites

• Node.js 18+
• VS Code 1.102.0+
• Glassbox CLI

Setup

# Clone the repository
git clone https://github.com/glassbox-ai/glassbox-vscode-extension.git
cd glassbox-vscode-extension

# Install dependencies
npm install

# Compile the extension
npm run compile

# Run tests
npm test

# Package the extension
npm run package

Development Commands

# Watch for changes
npm run watch

# Lint code
npm run lint

# Package extension
npm run package

# Publish extension
npm run publish

📚 Documentation

• Settings Documentation - Comprehensive settings guide
• Command Organization - Command structure and usage
• API Reference - Glassbox CLI documentation
• Examples - Sample projects and configurations

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

Code Style

• Follow TypeScript best practices
• Use ESLint for code linting
• Write comprehensive tests
• Update documentation as needed

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• VS Code Extension API
• OpenAI API
• Anthropic API
• Google AI
• Ollama

📞 Support

• Documentation: https://glassbox.ai/docs
• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Email: support@glassbox.ai