Skip to content
| Marketplace
Sign in
Visual Studio Code>Testing>GitHub Copilot Evals (Preview)New to Visual Studio Code? Get it now.
GitHub Copilot Evals (Preview)

GitHub Copilot Evals (Preview)

Preview

GitHub

github.com
|
22 installs
| (0) | Free
Inspect GitHub Copilot customizations and run evals from one VS Code pane.
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

GitHub Copilot Evals (Preview)

Create, run and compare tests for your Copilot skills and agents in VS Code. Turn a task into an eval, check the output, and keep the result to compare after your next change.

An eval is a repeatable test of AI work: give it a task, define what a good result looks like, and check what happened.

What you can test

  • Skills (SKILL.md): check whether a project skill or a skill in a Plugin source checkout produces the expected output and file changes.
  • Custom agents (*.agent.md): check whether a delegated agent completes a task and follows its instructions.
  • MCP tools: run file/link health checks or evaluate one local Node.js stdio MCP server per run (one task per fresh server start). Python MCP servers and remote MCP servers are not currently supported.

Install

VS Code editor: install GitHub Copilot Evals from the Visual Studio Marketplace, or search Extensions for GitHub.copilot-evals from publisher GitHub. The extension registers native VS Code Evals tools, sidebar views and Canvas, and bundles the six Evals Plugin skills without registering a duplicate MCP server.

VS Code Agents: install the standalone Evals Plugin (eval@github-copilot-evals) for the Copilot harness, which includes mcp.json (copilot-evals at ${PLUGIN_ROOT}/dist/mcp/server.mjs), the six Evals skills, automation blueprints and Canvas (native Agent Host Canvas and /eval commands activate when chat.canvases.enabled is enabled), starting the 15-tool copilot-evals MCP server for the launch workspace without cloning or manual .mcp.json configuration. The VS Code extension (GitHub.copilot-evals) also declares capabilities.agentsWindow.supported: true and runs natively in the Agents window (Customizations > Extensions > Supported), providing Evals tools, Command Palette actions, Webview Canvas (GitHub Copilot Evals: Open Evals Canvas), bundled skills and portable automation blueprints (automations/*.automation.md).

Use VS Code 1.138.0 or later on Windows, macOS or Linux. To try an available prerelease, choose Switch to Pre-Release Version in the extension's menu. Choose Switch to Release Version to return to stable.

For offline installation, use Extensions: Install from VSIX with the package matching your platform. Verify its SHA-256 against the checksum provided with that package.

Prerequisites

Before creating and running your first eval:

  1. Open a local project folder in desktop VS Code 1.138.0 or later and trust the folder when prompted.
  2. Sign in to GitHub Copilot in VS Code. Model-backed runs use VS Code's GitHub authentication; choose GitHub Copilot Evals: Change GitHub account to select the account for subsequent runs.
  3. Choose GitHub Copilot Evals: Set up test runner or open Evals Canvas > Settings > Advanced. Evals detects installed runners and offers guided installation.
Dependency Minimum Used for
Node.js 22.14.0 Local runner execution and checks
@microsoft/vally-cli 0.19.0 Eval validation and execution
Full GitHub Copilot CLI 1.0.85 Native custom-agent and MCP execution
Git 2.28.0 Native run isolation

To select an installation manually, use setup or configure copilotEvals.nodePath, copilotEvals.evaluatorPath and, when needed, copilotEvals.copilotPath. Project-level values are honored only after the host trusts that workspace; user-level VS Code settings remain separate from workspace trust. Configured workspace-level evaluator paths are not probed until explicitly approved in runner setup, even when they resolve to the declared direct project dependency. An auto-discovered direct project dependency remains supported; paths resolving into the workspace as arbitrary checkouts or aliases require explicit selection. An Advanced or setup-approved selection is remembered in machine-local state for the same workspace and exact resolved path; repository settings alone cannot grant that exception on another machine.

Evals uses your explicit runner selection first, then a compatible dependency in the selected project, then the runner on VS Code's launch PATH. It can also use a single compatible global installation. Setup lets you choose between multiple installations. Grader checks use the same verified runner selection.

Create your first eval

You can start from the GitHub Copilot Evals sidebar or from Copilot Chat:

  1. Select GitHub Copilot Evals in the Activity Bar. Under Assets, right-click a skill or agent and choose Create evals with Copilot (or enter /eval:create in Copilot Chat).

  2. Tell Copilot what a successful result should look like. For example, for a release-notes skill:

    Create an eval for this skill using one completed issue and one open issue.
    Check that the release notes include only completed work, preserve the
    issue IDs, and leave the input files unchanged.
    
  3. Review the proposed cases, input files and checks. Save the eval when they describe the behavior you want to test. Saving uses normal VS Code file permissions and never starts a run.

  4. Run the saved eval: under Evals > Project evals in the sidebar, click Run eval (the Play button) on the eval, or open the eval in Evals Canvas and choose Review & run, or enter /eval:run in Copilot Chat.

  5. Review the execution details and token/credit costs in the confirmation dialog, then approve the run.

  6. Choose Open result to see which checks passed and inspect the output, or choose Review conversation trace to read the captured prompts, answers and tool calls.

  7. After editing your skill or agent, run the eval again and use Compare saved runs (no AI calls) or /eval:compare to compare the new result against your earlier run.

Sidebar, bundled skills and Evals Canvas

Evals sidebar

Selecting GitHub Copilot Evals in the Activity Bar opens the workspace tree:

  • Evals groups Built-in baselines (extension-provided health and behavior checks) and Project evals (saved .eval.yaml definitions in your project).
  • Assets groups the skills, custom agents and MCP servers discovered in your project. Expand an asset to run its baselines or right-click it to choose Create evals with Copilot or Open source.
  • Files needing attention appears when discovery finds syntax, link or size problems in project files.
Task Where to start
Create tests for a skill or agent Create evals with Copilot in the sidebar or Command Palette
Check file formats and local references Run health checks in the sidebar toolbar
Test a skill or agent on a generated task Expand the asset under Assets and choose Run behavior baseline
Run a saved eval Run eval on an item under Project evals, or Review & run in Evals Canvas
Check your graders against expected answers Test graders, using a reference answer or saved conversation
Review or revise an eval Review eval with Copilot or Improve eval with Copilot
Revisit earlier results Previous runs in the sidebar toolbar or Command Palette
Compare previous results Compare saved runs (no AI calls) in the sidebar toolbar or Command Palette
Explore a skill comparison GitHub Copilot Evals: Compare in the Command Palette
Configure eval paths and test suites Evals Canvas > Settings > Project

Bundled Copilot Chat skills

In a trusted local desktop workspace with the extension installed, ask in plain language or use the six bundled Chat skills:

Skill Example request What it does and won't change
/eval:create Create an eval for this agent that checks its output files. Drafts new eval and fixture files from your asset, validates them, and saves under normal file permissions without overwriting existing files or running the eval.
/eval:review Review this eval and its saved results. Explains an eval definition or saved run result. Read-only: never edits files or starts a run.
/eval:improve Add a failure case to this eval and save a revised copy. Saves a revised copy alongside the original eval so the original remains intact; never runs the eval automatically.
/eval:run Run this eval. Prepares a saved eval from an open local project and requests separate execution approval before any model or tool calls run.
/eval:compare Compare these two saved runs. Opens the Compare view in Evals Canvas so you can compare saved runs or set up a skill comparison; never starts a run automatically.
/eval:setup Set up eval paths and test suites for this project. Inspects and proposes .vally.yaml folder and suite settings for review before saving; never installs tools or runs evals.

For /eval:run, a supplied eval name is matched against saved evals in open local projects. Without a name, or when a name needs clarification, choose from the project eval list. Preparing a selection does not approve execution; review the separate run approval before any model calls or tools run.

Evals Canvas

Run GitHub Copilot Evals: Open Evals Canvas from the Command Palette, click Open Evals Canvas in the sidebar toolbar, or right-click Project evals:

  • Create: draft and edit eval files and fixtures, check them with the local runner, and save reviewed files.
  • Visual, Outline and Source: select any saved eval in Canvas to inspect its task-to-check diagram (Visual), plain-language summary (Outline) or YAML definition (Source), then choose Review & run or Test graders.
  • Results: inspect completed, failed or cancelled runs, including check outcomes, outputs, errors, usage measurements and conversation traces.
  • Compare: compare two saved runs side by side without AI calls, or prepare a with-and-without skill comparison where each run is approved separately. Missing token or cost measurements are shown as unknown.
  • Settings: configure project eval folders and suites under Project, or verify runner executables and check retained locks under Advanced.

Guided authoring from Canvas

Use the Copilot box's Send to Copilot button to continue through the same skills used by Copilot Chat. Copilot reads the selected source, asks only consequential unanswered questions, and reviews and saves through the native authoring tools. Improve retains the original eval and saves a distinct candidate. This flow does not retrieve other sessions.

The handoff keeps the exact unsaved draft and request identity in private storage. Canvas's snapshot becomes read-only to avoid competing saves; it does not automatically receive the revised proposal. There is no separate conversation button or quick-assistant mode. Chat-imported drafts must finish in the original Chat or be reviewed and saved in Canvas, not sent back as a second conversation request. After a complete save in trusted local desktop VS Code, choose View saved eval in Canvas in the post-save notification to inspect the saved eval. Choose one if several were saved. Canvas refreshes and opens the saved details without replacing its retained draft or running anything. Run remains a separate choice with its existing approval. Closing the notification leaves you in chat. If Canvas is open in another project, keep its unfinished work and open the saved eval's project explicitly; the return action does not switch projects. VS Code opens Agent Chat using its native command; this is not a guarantee that a particular preexisting chat is selected. The bound project and request, not ambient chat context, determine the authoring scope. Unlike the App/CLI SDK, the current VS Code chat command has no verified separate display text or application-message provenance parameter. It still shows the full handoff query so required instructions are not lost. Compact handoff presentation in VS Code remains an explicit host capability gap.

An unconfirmed delivery retains the request without automatically resending it. Closing Canvas does not cancel work in Chat. Saving retains normal tool permissions; running still requires the separate cost-and-effects approval.

Runs, results, costs and safety

Saving an eval and running it are separate actions. File/link checks and reading or comparing saved results make no AI calls. AI authoring, review and behavior runs use Copilot models.

Each run asks you to review its execution effects and token/credit costs before approval. Eval setup, tools, custom graders and AI judges can execute code and incur charges. Review grader tests with the same care as agent runs. Runner installation has its own command-specific confirmation.

Native Run requests first check the local runner, inspect parameters and stage inputs for validation. This preparation does not authorize execution or installation. After validation, the single non-modal execution confirmation shows the shared run summary. A read-only review opens with exact inputs, effects, limits and version-aware historical costs. Declining or stopping at review saves a cancelled result; approval never carries over to another run. VS Code selects the run account only after approval and rechecks saved inputs and workspace trust before execution.

Use Stop to cancel active work. Work folders provide run isolation; workspace trust is still required for writes and execution, and execution is not an OS or network sandbox: runs have access to your machine's normal filesystem and network permissions. Per-run limits apply to individual runs, and total cost is not capped across runs.

A passing check doesn't guarantee the AI will get it right every time. Treat eval results and skill comparisons as evidence about the tested tasks and checks, and inspect the output and conversation trace alongside pass/fail counts.

Files and teamwork

  • Eval definitions and fixtures: new evals default to evals/ in your project root (with input data under evals/fixtures/<bundle>/), or use the eval directory configured in .vally.yaml. Commit eval definitions, fixtures and .vally.yaml to source control so your team can review and run the same checks.
  • Saved results and runner artifacts: saved run reports are written to .copilot\evals\results in your project, and grader test reports are written to .copilot\evals\grader-tests. If .vally.yaml sets paths.results, raw runner output uses that directory. Keep result directories out of source control or review their contents carefully before sharing, because saved reports and traces can contain prompts, model outputs, tool outputs and local project data.

Supported hosts

Host Experience
VS Code editor, trusted local desktop project Evals sidebar, bundled Plugin, Canvas, local runs and saved reports
Remote workspace extension host Native Run using the workspace's runner configuration and saved-result comparison
Browser workspace Read saved reports through the workspace file provider
VS Code Agents, Copilot harness Evals VS Code extension (GitHub.copilot-evals) and/or standalone Evals Plugin (eval@github-copilot-evals) with bundled copilot-evals MCP server (mcp.json), six skills, automation blueprints and Canvas

Open remote reports through the workspace extension host. Browser hosts can read saved reports through their workspace provider; reports identify when their original local workspace binding cannot be verified.

Troubleshooting

Problem Action
Runner missing or incompatible Run GitHub Copilot Evals: Set up test runner.
Project files missing from the sidebar Check ignore rules, fix any discovery diagnostics, then use Refresh.
Need run diagnostics Open Output > GitHub Copilot Evals.
Duplicate Evals panes after an upgrade Uninstall copilot-evals.copilot-evals or copilot-evals-local.copilot-evals, reload VS Code, and install GitHub.copilot-evals.
A retained lock blocks the next run Use GitHub Copilot Evals: Check a retained run lock, review the recorded process, then use Recover a retained run lock after review. Preserve the lock while cleanup is uncertain.

Report bugs and suggestions in GitHub Copilot Evals. Include your extension and VS Code versions, OS, reproduction steps and sanitized diagnostics. For security reports, follow the security policy.

Telemetry and privacy

The VS Code extension follows VS Code's telemetry.telemetryLevel setting. Set it to off to stop extension telemetry; error allows error events but not usage events. When enabled, extension events cover component activation, Evals workflow requests, approval dispositions, outcomes and error categories, with bounded duration and trial-count measurements. Standard VS Code SDK fields include a pseudonymous machine identifier and a session identifier. Events may also include an optional pseudonymous cross-tool developer device identifier. Events exclude source code, prompts, model outputs, project paths, repository names and raw error messages.

The bundled Plugin is a separate component with separate telemetry behavior. When the Plugin identifies an editor host, it does not send its own telemetry without an explicit host-consent bridge; no such bridge is currently implemented. Standalone Plugin telemetry has separate environment opt-outs: set any one of COPILOT_EVALS_TELEMETRY=off, DO_NOT_TRACK=1 or OTEL_SDK_DISABLED=true before starting its host. These do not replace the VS Code extension's telemetry.telemetryLevel setting. See Plugin telemetry details.

AI authoring, review and execution can send selected inputs to the configured provider under their separate permissions. Saved run artifacts remain local and can contain those inputs and outputs.

See VS Code telemetry documentation, the GitHub Privacy Statement and Microsoft Privacy Statement.

License

Licensed under the GitHub License Terms for Extensions. See the LICENSE file included with the extension.

  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
  • Your Privacy Choices
  • Consumer Health Privacy
© 2026 Microsoft