GitHub Copilot Evals (Preview)

Create, run and compare tests for your Copilot skills and agents in VS Code. Turn a task into an eval, check the output, and keep the result to compare after your next change.

Install

VS Code editor: install GitHub Copilot Evals from the Visual Studio Marketplace, or search Extensions for GitHub.copilot-evals from publisher GitHub. The extension registers native VS Code Evals tools, sidebar views and Canvas, and bundles the six Evals Plugin skills without registering a duplicate MCP server.

VS Code Agents: install the standalone Evals Plugin (eval@github-copilot-evals) for the Copilot harness. The standalone Plugin includes mcp.json (copilot-evals at ${PLUGIN_ROOT}/dist/mcp/server.mjs), the six Evals skills and Canvas, so Agent Plugins 1.0 hosts automatically start the 20-tool copilot-evals MCP server for the launch workspace without cloning or manual .mcp.json configuration.

Use VS Code 1.138.0 or later on Windows, macOS or Linux. To try an available prerelease, choose Switch to Pre-Release Version in the extension's menu. Choose Switch to Release Version to return to stable.

For offline installation, use Extensions: Install from VSIX with the package matching your platform. Verify its SHA-256 against the checksum provided with that package.

Create your first eval

Open a local project, trust the folder and sign in to GitHub Copilot.
Select GitHub Copilot Evals in the Activity Bar. Find a skill or agent, right-click it and choose Create evals with Copilot.

Tell Copilot what a successful result should look like. For example, for a release-notes skill:

Create an eval for this skill using one completed issue and one open issue.
Check that the release notes include only completed work, preserve the
issue IDs, and leave the input files unchanged.

Review the proposed cases, input files and checks. Save the eval when they describe the behavior you want to test.
Choose Review & run. Review the execution details and token/credit costs, then approve the run. Evals guides you through runner setup if needed.
Open the result to see which checks passed, inspect the output and read the conversation trace.

For an existing eval, select it under Project evals and use its Play action. Use Previous runs to revisit earlier results.

What you can do

Task	Where to start
Create tests for a skill or agent	Create evals with Copilot in the sidebar or Command Palette
Check file formats and local references	Run health checks in the sidebar toolbar
Test a skill or agent on a generated task	Expand the asset and choose a behavior baseline
Check your graders against expected answers	Test graders, using a reference answer or saved conversation
Review or revise an eval	Review eval with Copilot or Improve eval with Copilot
Compare previous results	Compare saved runs (no AI calls) in the Command Palette
Explore a skill comparison	GitHub Copilot Evals: Compare
Configure the project or a CI workflow	Evals Canvas > Settings > Project or CI pipeline

Improve saves a revised copy so you can keep the original eval. For MCP tools, use file/link checks or create an eval for a specific tool action.

Use Copilot Chat

In a trusted local desktop workspace with the extension installed, ask in plain language or use the bundled Chat skills:

Skill	Example request
`/eval:create`	Create an eval for this agent that checks its output files.
`/eval:review`	Review this eval and its saved results.
`/eval:improve`	Add a failure case to this eval and save a revised copy.
`/eval:run`	Run this eval.
`/eval:compare`	Compare these two saved runs.
`/eval:setup`	Set up evals and a CI workflow for this project.

Explore evals and results in Canvas

Run GitHub Copilot Evals: Open Evals Canvas from the Command Palette. Use Create to work on drafts, or select a saved eval to inspect its Visual, Outline or Source view.

Open Results to inspect completed runs. Reports include check outcomes, output, errors and available usage measurements. Review conversation trace shows captured requests, answers and tool calls.

Use Compare to inspect differences between saved runs or prepare a skill comparison. It keeps the original results available for inspection. Missing token or cost measurements are shown as unknown.

Results and runner artifacts are saved under .copilot\evals\results in your project. Review their contents before sharing them.

Runner setup

Choose GitHub Copilot Evals: Set up test runner or open Evals Canvas > Settings > Advanced. Evals detects installed runners and offers guided installation.

Dependency	Minimum	Used for
Node.js	22.14.0	Local runner execution and checks
`@microsoft/vally-cli`	0.19.0	Eval validation and execution
Full GitHub Copilot CLI	1.0.85	Native custom-agent and MCP execution
Git	2.28.0	Native run isolation

To select an installation manually, use setup or configure copilotEvals.nodePath, copilotEvals.evaluatorPath and, when needed, copilotEvals.copilotPath.

Evals uses your explicit runner selection first, then a compatible dependency in the selected project, then the runner on VS Code's launch PATH. It can also use a single compatible global installation. Setup lets you choose between multiple installations.

Model-backed runs use VS Code's GitHub authentication. Choose GitHub Copilot Evals: Change GitHub account to select the account for subsequent runs.

Runs and costs

Each run asks you to review its execution effects and token/credit costs before approval. Eval setup, tools, custom graders and AI judges can execute code and incur charges. Review grader tests with the same care as agent runs. Runner installation has its own command-specific confirmation.

Native Run requests first check the local runner, inspect parameters and stage inputs for validation. This preparation does not authorize execution or installation. After validation, the single non-modal execution confirmation shows the shared run summary. A read-only review opens with exact inputs, effects, limits and version-aware historical costs. Declining or stopping at review saves a cancelled result; approval never carries over to another run. VS Code selects the run account only after approval and rechecks saved inputs and workspace trust before execution.

File/link checks and reading or comparing saved results make no AI calls. AI authoring, review and behavior runs use Copilot models. Saving an eval and running it are separate actions.

Use Stop to cancel active work. Work folders provide run isolation; workspace trust is still required for writes and execution, and execution has access to your machine's normal filesystem and network permissions.

Supported hosts

Host	Experience
VS Code editor, trusted local desktop project	Evals sidebar, bundled Plugin, Canvas, local runs and saved reports
Remote workspace extension host	Native Run using the workspace's runner configuration and saved-result comparison
Browser workspace	Read saved reports through the workspace file provider
VS Code Agents, Copilot harness	Standalone Evals Plugin (`eval@github-copilot-evals`) with bundled `copilot-evals` MCP server (`mcp.json`), six skills and Canvas

Open remote reports through the workspace extension host. Browser hosts can read saved reports through their workspace provider; reports identify when their original local workspace binding cannot be verified.

Troubleshooting

Problem	Action
Runner missing or incompatible	Run GitHub Copilot Evals: Set up test runner.
Project files missing from the sidebar	Check ignore rules, fix any discovery diagnostics, then use Refresh.
Need run diagnostics	Open Output > GitHub Copilot Evals.
Duplicate Evals panes after an upgrade	Uninstall `copilot-evals.copilot-evals` or `copilot-evals-local.copilot-evals`, reload VS Code, and install `GitHub.copilot-evals`.
A retained lock blocks the next run	Use GitHub Copilot Evals: Check a retained run lock, review the recorded process, then use Recover a retained run lock after review. Preserve the lock while cleanup is uncertain.

Report bugs and suggestions in GitHub Copilot Evals. Include your extension and VS Code versions, OS, reproduction steps and sanitized diagnostics. For security reports, follow the security policy.

Telemetry and privacy

The VS Code extension follows VS Code's telemetry.telemetryLevel setting. Set it to off to stop extension telemetry; error allows error events but not usage events. When enabled, extension events cover component activation, Evals workflow requests, approval dispositions, outcomes and error categories, with bounded duration and trial-count measurements. Standard VS Code SDK fields include a pseudonymous machine identifier and a session identifier. Events may also include an optional pseudonymous cross-tool developer device identifier. Events exclude source code, prompts, model outputs, project paths, repository names and raw error messages.

The bundled Plugin is a separate component with separate telemetry behavior. When the Plugin identifies an editor host, it does not send its own telemetry without an explicit host-consent bridge; no such bridge is currently implemented. Standalone Plugin telemetry has separate environment opt-outs: set any one of COPILOT_EVALS_TELEMETRY=off, DO_NOT_TRACK=1 or OTEL_SDK_DISABLED=true before starting its host. These do not replace the VS Code extension's telemetry.telemetryLevel setting. See Plugin telemetry details.

AI authoring, review and execution can send selected inputs to the configured provider under their separate permissions. Saved run artifacts remain local and can contain those inputs and outputs.

See VS Code telemetry documentation, the GitHub Privacy Statement and Microsoft Privacy Statement.

License

Licensed under the GitHub License Terms for Extensions. See the LICENSE file included with the extension.