Skill Evaluator

AI-powered skill evaluation for VS Code. Use any LLM - free or your own.

Requirements

• VS Code 1.50+
• Python 3.7+ installed and available in PATH

Python Dependencies (Auto-Installed)

The extension automatically installs required Python packages on first run:

• requests - HTTP library for LLM API calls
• openai - Azure OpenAI support
• python-dotenv - Environment variable loading

✨ No manual setup needed! The extension handles dependency installation automatically.

If auto-install fails, manually run:

pip install requests openai python-dotenv

Prerequisites: Skill & Test Cases

Before evaluating, you need:

1. Skill File (.md)

A markdown file with these 7 required sections:

• name: (YAML front matter)
• description: (YAML)
• version: (YAML)
• # Goal (Markdown section)
• # Inputs (Markdown section)
• # Workflow (Markdown section)
• # Output (Markdown section)

Example: skills/book-recommendation.md

2. Test Cases File (.json)

A JSON file with test cases for your skill. Must contain:

• test_cases array with user prompts and expected outcomes

Example: test_cases/book-recommendation.json

{
  "test_cases": [
    {
      "user_prompt": "Recommend a sci-fi book for beginners",
      "expected_outcome": "Skill recommends age-appropriate sci-fi book with reasoning"
    },
    {
      "user_prompt": "Suggest a mystery novel",
      "expected_outcome": "Skill recommends mystery book with plot summary"
    }
  ]
}

File Location: The evaluator looks for test cases in a test_cases/ folder with the same name as your skill file.

Install

1. Download: directai-skill-evaluator-1.0.0.vsix
2. VS Code → Ctrl+Shift+X → Install from VSIX
3. Done!

Configure (Pick ONE)

Option A: FREE (Groq)

1. Get free key: https://console.groq.com/keys
2. VS Code Settings → Search "groqApiKey" → Paste your key
3. Done! (Works for single file AND batch evaluation)

Option B: Your LLM (Optional Primary)

VS Code Settings:
  skillEvaluator.llmUrl    = your-api-endpoint
  skillEvaluator.llmApiKey = your-api-key
  skillEvaluator.llmModel  = model-name

How to Use

Evaluate Single File

1. Open a SKILL.md file in VS Code
2. Press Ctrl+Shift+P (Command Palette)
3. Type: Evaluate Single File
4. Press Enter
5. Wait 10-30 seconds
6. Check Output panel (Ctrl+Shift+U) for validation + summary results

Quick Shortcut: Ctrl+Shift+Alt+E

Evaluate Multiple Files

1. Press Ctrl+Shift+P (Command Palette)
2. Type: Evaluate Multiple Files
3. Press Enter
4. Select folder with skill files
5. Results shown in Output panel
6. HTML reports saved to evaluation reports folder

What You'll See

Console Output (Simple & Clean)

[INFO] Single File: No primary LLM configured - checking for Groq fallback...
✓ Using Groq (free) as fallback model
======================================================================

======================================================================
[VALIDATION] SKILL STRUCTURE CHECK
======================================================================

REQUIRED SECTIONS:
  [OK] Skill Name................................... [OK]
  [OK] Description.................................. [OK]
  [OK] Version...................................... [OK]
  [OK] Goal......................................... [OK]
  [OK] Inputs....................................... [OK]
  [OK] Workflow..................................... [OK]
  [OK] Output....................................... [OK]

Status: 7/7 required sections present

[PASS] All required sections found
[READY FOR EVALUATION]

======================================================================
SUMMARY
======================================================================

Total Tests: 5
Passed: 4 | Partial: 1 | Failed: 0
Average Completeness: 85%

[READY] - Skill is production-ready

HTML Report (Full Details)

Each evaluation generates an HTML report with:

• Skill Metadata: Model used, timestamp, test files
• Summary Table: Test counts and completeness metrics
• Validation Results: All required sections checked
• Test Results: Individual test scores
• Identified Gaps: What's missing from the skill
• Recommendations: Improvements by category:
  • Description Improvements
  • Capabilities to Expand
  • Scope Clarification

Console vs Report

Console Output:

• ✅ LLM provider detection (Groq fallback or primary LLM)
• ✅ Validation check results (all 7 sections)
• ✅ Summary (test counts, completeness %)
• ✅ Verdict (READY / NEEDS WORK / NOT READY)

NOT in Console (All in HTML Report):

• ❌ Individual test details
• ❌ LLM debug logs (prompts, headers, etc.)
• ❌ Gaps and recommendations
• ❌ Full test analysis

View Logs

Press Ctrl+Shift+U and select "Skill Evaluator" dropdown to see detailed logs.

Validation

Every skill file is checked for 7 required sections:

1. name: (YAML)
2. description: (YAML)
3. version: (YAML)
4. Goal (Markdown section)
5. Inputs (Markdown section)
6. Workflow (Markdown section)
7. Output (Markdown section)

Batch evaluation skips files that fail validation.

Supported Providers

• Groq (free - llama-3.1-8b-instant)
• Azure OpenAI (custom headers supported)
• OpenAI
• Together AI
• Any OpenAI-compatible endpoint

Troubleshooting

"ModuleNotFoundError" → Extension auto-installs dependencies on first run → If auto-install fails: pip install requests openai python-dotenv → Restart VS Code after installing

"No LLM configured" → Set Groq key OR primary LLM in settings

"401 Unauthorized" → Check API key (no extra spaces, no "Bearer " prefix)

Batch evaluation fails → Check if Groq key is set (used as fallback)

"Validation Failed" → Add missing required sections to skill file

"Timeout" → Some evaluations take 30-60 seconds, be patient

"Python not found" → Ensure Python 3.7+ is in PATH. On Windows, check installation wizard selected "Add Python to PATH"

Dependencies still not installing? → Check Python version: python --version

Version

1.0.0 - Production ready with Groq fallback, validation, and detailed HTML reports