WinAI Eval Platform Skills Extension

VS Code extension that provides Agent Skills for the WinAI Eval Platform. Enables AI assistants (GitHub Copilot, Claude) to call Eval Platform REST APIs directly.

What This Does

This extension teaches AI assistants how to call the Eval Platform REST API directly using PowerShell and Azure CLI tokens. The skills are loaded on-demand by the AI assistant when relevant to your request.

Included Skills

Skill	Description
winaieval-skills-setup	Set up authentication and prerequisites — Azure CLI login, token verification, API connectivity test
winaieval-api	Complete REST API reference — projects, agents, datasets, experiments, runs, Windows images, module versioning, run logs/statistics/comparison
winaieval-agent-development-and-evaluation	End-to-end workflow — develop an agent locally (Phase 1) → upload & benchmark on the Eval Platform (Phase 2), with troubleshooting & debugging playbook

Prerequisites

VS Code ≥ 1.99 with GitHub Copilot
Azure CLI — az --version
PowerShell 7+ — pwsh --version
Eval Platform access — App Role + project membership

Quick Start

Install the extension
Open Copilot Chat and invoke a skill using either method below:

Option A: Natural Language (Auto-Discovery)

Simply describe what you need — Copilot matches the right skill automatically:

"Help me set up Eval Platform authentication" → triggers winaieval-skills-setup
"I want to develop and evaluate an agent" → triggers winaieval-agent-development-and-evaluation
"List all projects via the Eval Platform API" → triggers winaieval-api

Option B: Direct Invocation

Type the skill name directly in Copilot Chat to invoke it explicitly:

/winaieval-skills-setup — set up auth & prerequisites
/winaieval-agent-development-and-evaluation — end-to-end agent workflow
/winaieval-api — call any Eval Platform REST API

How It Works

Skills use progressive disclosure:

Discovery — Copilot sees skill names & descriptions (lightweight)
Loading — When relevant, the SKILL.md instructions are loaded into context
Execution — The AI assistant runs PowerShell commands to call REST APIs

Authentication flows through Azure CLI (az account get-access-token), so no API keys are stored in chat or config files.

Development

To modify skills:

Edit the SKILL.md files in skills/*/
Rebuild: npx @vscode/vsce package --no-dependencies
Reinstall the VSIX

Or during development, add the skills directory to your workspace settings:

{
  "chat.agentSkillsLocations": [
    "${workspaceFolder}/winaieval-skills-extension/skills"
  ]
}

WinAI Eval Platform Skills

WinAI Eval Team