Skip to content
| Marketplace
Sign in
Visual Studio Code>Programming Languages>Chat Customizations EvaluationsNew to Visual Studio Code? Get it now.
Chat Customizations Evaluations

Chat Customizations Evaluations

Microsoft

microsoft.com
|
425,328 installs
| (13) | Free
Extension for analyzing, validating, and improving AI prompts
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

Chat Customizations Evaluations

License: MIT

An extension for analyzing and improving AI prompt files. Works with .prompt.md, .agent.md, SKILL.md, and .instructions.md files — providing LLM-powered semantic analysis directly in VS Code.

Features

LLM-Powered Analysis (via GitHub Copilot)

  • Contradiction Detection — Finds logical, behavioral, and format conflicts
  • Semantic Ambiguity — Ambiguity analysis with rewrite suggestions
  • Persona Consistency — Detects conflicting personality traits and tone drift
  • Cognitive Load Assessment — Warns about overly complex prompts with too many nested conditions
  • Semantic Coverage — Identifies gaps in intent handling and missing error paths
  • Composition Conflict Analysis — Detects conflicts between a prompt and other prompt files it imports via markdown links

Waza Integration

  • Create Eval Scaffold — Generates eval files for a skill via waza new eval <skill-name>
  • Run Evaluation — Executes skill evaluation via waza run <eval.yaml> --context-dir <skill-dir>
  • Automatic Local Fallback — If waza is not on PATH, commands attempt a local fallback via go run ./cmd/waza when a sibling waza repo is available

Editor Integration

  • Editor Title Bar — Analyze Prompt button appears when editing prompt files
  • Command Palette — Chat Customizations Evaluations: Analyze Prompt command
  • Problems Panel — All diagnostics appear in the standard VS Code Problems panel with precise line and column locations

Supported File Types

Pattern Type
*.prompt.md Prompt
*.agent.md Agent
*.instructions.md Instructions

Usage

  1. Open any supported prompt file in VS Code
  2. Run Chat Customizations Evaluations: Analyze Prompt from the command palette or click the beaker icon in the editor title bar
  3. View results in the Problems panel (Ctrl+Shift+M / Cmd+Shift+M)

LLM analysis requires GitHub Copilot — no API keys needed. Just sign in to GitHub Copilot in VS Code.

Commands

Command Description
Chat Customizations Evaluations: Analyze Prompt Run full LLM-powered analysis on the active file
Chat Customizations Evaluations: Create Waza Eval Scaffold Create eval.yaml and task files for the active skill
Chat Customizations Evaluations: Run Waza Evaluation Run the skill's eval suite
Chat Customizations Evaluations: Download Waza Binary Download the latest platform-specific waza binary and configure the extension to use it

Configuration

Setting Default Description
chatCustomizationsEvaluations.enable true Enable/disable the extension
chatCustomizationsEvaluations.trace.server off Trace communication between VS Code and the language server
chatCustomizationsEvaluations.customDiagnostics [] Array of custom diagnostic objects with name and description fields
chatCustomizationsEvaluations.waza.command waza Command used to run waza (for example /usr/local/bin/waza)

License

MIT

  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
© 2026 Microsoft