GitHub Copilot Evals (Preview)
Create, run and compare tests for your Copilot skills and agents in VS Code.
Turn a task into an eval, check the output, and keep the result to compare
after your next change.
An eval is a repeatable test of AI work: give it a task, define what a good
result looks like, and check what happened.
What you can test
- Skills (
SKILL.md): check whether a project skill or a skill in a Plugin
source checkout produces the expected output and file changes.
- Custom agents (
*.agent.md): check whether a delegated agent completes a
task and follows its instructions.
- MCP tools: run file/link health checks or evaluate one local Node.js stdio
MCP server per run (one task per fresh server start). Python MCP servers and
remote MCP servers are not currently supported.
Install
VS Code editor: install GitHub Copilot Evals from the Visual Studio Marketplace,
or search Extensions for GitHub.copilot-evals from publisher GitHub.
The extension registers native VS Code Evals tools, sidebar views and Canvas,
and bundles the six Evals Plugin skills without registering a duplicate MCP
server.
VS Code Agents: install the standalone Evals Plugin
(eval@github-copilot-evals) for the Copilot harness, which includes mcp.json
(copilot-evals at ${PLUGIN_ROOT}/dist/mcp/server.mjs), the six Evals skills,
automation blueprints and Canvas (native Agent Host Canvas and /eval commands
activate when chat.canvases.enabled is enabled), starting the 15-tool
copilot-evals MCP server for the launch workspace without cloning or manual
.mcp.json configuration. The VS Code extension
(GitHub.copilot-evals) also declares capabilities.agentsWindow.supported: true
and runs natively in the Agents window (Customizations > Extensions > Supported),
providing Evals tools, Command Palette actions, Webview Canvas
(GitHub Copilot Evals: Open Evals Canvas), bundled skills and portable
automation blueprints (automations/*.automation.md).
Use VS Code 1.138.0 or later on Windows, macOS or Linux. To try an available
prerelease, choose Switch to Pre-Release Version in the extension's menu.
Choose Switch to Release Version to return to stable.
For offline installation, use Extensions: Install from VSIX with the
package matching your platform. Verify its SHA-256 against the checksum
provided with that package.
Prerequisites
Before creating and running your first eval:
- Open a local project folder in desktop VS Code 1.138.0 or later and
trust the folder when prompted.
- Sign in to GitHub Copilot in VS Code. Model-backed runs use VS Code's GitHub
authentication; choose GitHub Copilot Evals: Change GitHub account to
select the account for subsequent runs.
- Choose GitHub Copilot Evals: Set up test runner or open
Evals Canvas > Settings > Advanced. Evals detects installed runners and
offers guided installation.
| Dependency |
Minimum |
Used for |
| Node.js |
22.14.0 |
Local runner execution and checks |
@microsoft/vally-cli |
0.19.0 |
Eval validation and execution |
| Full GitHub Copilot CLI |
1.0.85 |
Native custom-agent and MCP execution |
| Git |
2.28.0 |
Native run isolation |
To select an installation manually, use setup or configure
copilotEvals.nodePath, copilotEvals.evaluatorPath and, when needed,
copilotEvals.copilotPath.
Project-level values are honored only after the host trusts that workspace;
user-level VS Code settings remain separate from workspace trust. Configured
workspace-level evaluator paths are not probed until explicitly approved in
runner setup, even when they resolve to the declared direct project dependency.
An auto-discovered direct project dependency remains supported; paths resolving
into the workspace as arbitrary checkouts or aliases require explicit
selection. An Advanced or setup-approved selection is remembered in
machine-local state for the same workspace and exact resolved path; repository
settings alone cannot grant that exception on another machine.
Evals uses your explicit runner selection first, then a compatible dependency
in the selected project, then the runner on VS Code's launch PATH. It can also
use a single compatible global installation. Setup lets you choose between
multiple installations. Grader checks use the same verified runner selection.
Create your first eval
You can start from the GitHub Copilot Evals sidebar or from Copilot Chat:
Select GitHub Copilot Evals in the Activity Bar. Under Assets, right-click
a skill or agent and choose Create evals with Copilot (or enter /eval:create
in Copilot Chat).
Tell Copilot what a successful result should look like. For example, for
a release-notes skill:
Create an eval for this skill using one completed issue and one open issue.
Check that the release notes include only completed work, preserve the
issue IDs, and leave the input files unchanged.
Review the proposed cases, input files and checks. Save the eval when they
describe the behavior you want to test. Saving uses normal VS Code file
permissions and never starts a run.
Run the saved eval: under Evals > Project evals in the sidebar, click
Run eval (the Play button) on the eval, or open the eval in Evals Canvas
and choose Review & run, or enter /eval:run in Copilot Chat.
Review the execution details and token/credit costs in the confirmation dialog,
then approve the run.
Choose Open result to see which checks passed and inspect the output, or
choose Review conversation trace to read the captured prompts, answers
and tool calls.
After editing your skill or agent, run the eval again and use
Compare saved runs (no AI calls) or /eval:compare to compare the new
result against your earlier run.
Selecting GitHub Copilot Evals in the Activity Bar opens the workspace tree:
- Evals groups Built-in baselines (extension-provided health and behavior
checks) and Project evals (saved
.eval.yaml definitions in your project).
- Assets groups the skills, custom agents and MCP servers discovered in your
project. Expand an asset to run its baselines or right-click it to choose
Create evals with Copilot or Open source.
- Files needing attention appears when discovery finds syntax, link or size
problems in project files.
| Task |
Where to start |
| Create tests for a skill or agent |
Create evals with Copilot in the sidebar or Command Palette |
| Check file formats and local references |
Run health checks in the sidebar toolbar |
| Test a skill or agent on a generated task |
Expand the asset under Assets and choose Run behavior baseline |
| Run a saved eval |
Run eval on an item under Project evals, or Review & run in Evals Canvas |
| Check your graders against expected answers |
Test graders, using a reference answer or saved conversation |
| Review or revise an eval |
Review eval with Copilot or Improve eval with Copilot |
| Revisit earlier results |
Previous runs in the sidebar toolbar or Command Palette |
| Compare previous results |
Compare saved runs (no AI calls) in the sidebar toolbar or Command Palette |
| Explore a skill comparison |
GitHub Copilot Evals: Compare in the Command Palette |
| Configure eval paths and test suites |
Evals Canvas > Settings > Project |
Bundled Copilot Chat skills
In a trusted local desktop workspace with the extension installed, ask in plain
language or use the six bundled Chat skills:
| Skill |
Example request |
What it does and won't change |
/eval:create |
Create an eval for this agent that checks its output files. |
Drafts new eval and fixture files from your asset, validates them, and saves under normal file permissions without overwriting existing files or running the eval. |
/eval:review |
Review this eval and its saved results. |
Explains an eval definition or saved run result. Read-only: never edits files or starts a run. |
/eval:improve |
Add a failure case to this eval and save a revised copy. |
Saves a revised copy alongside the original eval so the original remains intact; never runs the eval automatically. |
/eval:run |
Run this eval. |
Prepares a saved eval from an open local project and requests separate execution approval before any model or tool calls run. |
/eval:compare |
Compare these two saved runs. |
Opens the Compare view in Evals Canvas so you can compare saved runs or set up a skill comparison; never starts a run automatically. |
/eval:setup |
Set up eval paths and test suites for this project. |
Inspects and proposes .vally.yaml folder and suite settings for review before saving; never installs tools or runs evals. |
For /eval:run, a supplied eval name is matched against saved evals in open
local projects. Without a name, or when a name needs clarification, choose from
the project eval list. Preparing a selection does not approve execution; review
the separate run approval before any model calls or tools run.
Evals Canvas
Run GitHub Copilot Evals: Open Evals Canvas from the Command Palette, click
Open Evals Canvas in the sidebar toolbar, or right-click Project evals:
- Create: draft and edit eval files and fixtures, check them with the local
runner, and save reviewed files.
- Visual, Outline and Source: select any saved eval in Canvas to
inspect its task-to-check diagram (Visual), plain-language summary
(Outline) or YAML definition (Source), then choose Review & run or
Test graders.
- Results: inspect completed, failed or cancelled runs, including check
outcomes, outputs, errors, usage measurements and conversation traces.
- Compare: compare two saved runs side by side without AI calls, or prepare
a with-and-without skill comparison where each run is approved separately.
Missing token or cost measurements are shown as unknown.
- Settings: configure project eval folders and suites under Project, or
verify runner executables and check retained locks under Advanced.
Guided authoring from Canvas
Use the Copilot box's Send to Copilot button to continue
through the same skills used by Copilot Chat. Copilot reads the selected
source, asks only consequential unanswered questions, and reviews and saves
through the native authoring tools. Improve retains the original eval and
saves a distinct candidate. This flow does not retrieve other sessions.
The handoff keeps the exact unsaved draft and request identity in private
storage. Canvas's snapshot becomes read-only to avoid competing saves; it
does not automatically receive the revised proposal. There is no separate
conversation button or quick-assistant mode. Chat-imported drafts must finish
in the original Chat or be reviewed and saved in Canvas, not sent back as a
second conversation request.
After a complete save in trusted local desktop VS Code, choose View saved eval
in Canvas in the post-save notification to inspect the saved eval. Choose one
if several were saved. Canvas refreshes and opens the saved details without
replacing its retained draft or running anything. Run remains a separate
choice with its existing approval. Closing the notification leaves you in chat.
If Canvas is open in another project, keep its unfinished work and open the
saved eval's project explicitly; the return action does not switch projects.
VS Code opens Agent Chat using its native command; this is not a guarantee
that a particular preexisting chat is selected. The bound project and request,
not ambient chat context, determine the authoring scope.
Unlike the App/CLI SDK, the current VS Code chat command has no verified
separate display text or application-message provenance parameter. It still
shows the full handoff query so required instructions are not lost. Compact
handoff presentation in VS Code remains an explicit host capability gap.
An unconfirmed delivery retains the request without automatically resending
it. Closing Canvas does not cancel work in Chat. Saving retains normal tool
permissions; running still requires the separate cost-and-effects approval.
Runs, results, costs and safety
Saving an eval and running it are separate actions. File/link checks and reading
or comparing saved results make no AI calls. AI authoring, review and behavior
runs use Copilot models.
Each run asks you to review its execution effects and token/credit costs before
approval. Eval setup, tools, custom graders and AI judges can execute code and
incur charges. Review grader tests with the same care as agent runs.
Runner installation has its own command-specific confirmation.
Native Run requests first check the local runner, inspect parameters and stage
inputs for validation. This preparation does not authorize execution or installation.
After validation, the single non-modal execution confirmation shows the shared
run summary. A read-only review opens with exact inputs, effects, limits and
version-aware historical costs. Declining or stopping at review saves a cancelled result;
approval never carries over to another run. VS Code selects the run account only
after approval and rechecks saved inputs and workspace trust before execution.
Use Stop to cancel active work. Work folders provide run isolation;
workspace trust is still required for writes and execution, and execution
is not an OS or network sandbox: runs have access to your machine's normal
filesystem and network permissions. Per-run limits apply to individual runs,
and total cost is not capped across runs.
A passing check doesn't guarantee the AI will get it right every time. Treat
eval results and skill comparisons as evidence about the tested tasks and checks,
and inspect the output and conversation trace alongside pass/fail counts.
Files and teamwork
- Eval definitions and fixtures: new evals default to
evals/ in your
project root (with input data under evals/fixtures/<bundle>/), or use the
eval directory configured in .vally.yaml. Commit eval definitions, fixtures
and .vally.yaml to source control so your team can review and run the same
checks.
- Saved results and runner artifacts: saved run reports are written to
.copilot\evals\results in your project, and grader test reports are written
to .copilot\evals\grader-tests. If .vally.yaml sets paths.results, raw
runner output uses that directory. Keep result directories out of source
control or review their contents carefully before sharing, because saved
reports and traces can contain prompts, model outputs, tool outputs and local
project data.
Supported hosts
| Host |
Experience |
| VS Code editor, trusted local desktop project |
Evals sidebar, bundled Plugin, Canvas, local runs and saved reports |
| Remote workspace extension host |
Native Run using the workspace's runner configuration and saved-result comparison |
| Browser workspace |
Read saved reports through the workspace file provider |
| VS Code Agents, Copilot harness |
Evals VS Code extension (GitHub.copilot-evals) and/or standalone Evals Plugin (eval@github-copilot-evals) with bundled copilot-evals MCP server (mcp.json), six skills, automation blueprints and Canvas |
Open remote reports through the workspace extension host.
Browser hosts can read saved reports through their workspace provider;
reports identify when their original local workspace binding cannot be verified.
Troubleshooting
| Problem |
Action |
| Runner missing or incompatible |
Run GitHub Copilot Evals: Set up test runner. |
| Project files missing from the sidebar |
Check ignore rules, fix any discovery diagnostics, then use Refresh. |
| Need run diagnostics |
Open Output > GitHub Copilot Evals. |
| Duplicate Evals panes after an upgrade |
Uninstall copilot-evals.copilot-evals or copilot-evals-local.copilot-evals, reload VS Code, and install GitHub.copilot-evals. |
| A retained lock blocks the next run |
Use GitHub Copilot Evals: Check a retained run lock, review the recorded process, then use Recover a retained run lock after review. Preserve the lock while cleanup is uncertain. |
Report bugs and suggestions in GitHub Copilot Evals.
Include your extension and VS Code versions, OS, reproduction steps and
sanitized diagnostics. For security reports, follow the
security policy.
Telemetry and privacy
The VS Code extension follows VS Code's telemetry.telemetryLevel setting.
Set it to off to stop extension telemetry; error allows error events but
not usage events. When enabled, extension events cover component activation,
Evals workflow requests, approval dispositions, outcomes and error categories,
with bounded duration and trial-count measurements. Standard VS Code SDK
fields include a pseudonymous machine identifier and a session identifier.
Events may also include an optional pseudonymous cross-tool developer device
identifier. Events exclude source code, prompts, model outputs, project paths,
repository names and raw error messages.
The bundled Plugin is a separate component with separate telemetry behavior.
When the Plugin identifies an editor host, it does not send its own telemetry
without an explicit host-consent bridge; no such bridge is currently
implemented. Standalone Plugin telemetry has separate environment opt-outs:
set any one of COPILOT_EVALS_TELEMETRY=off, DO_NOT_TRACK=1 or
OTEL_SDK_DISABLED=true before starting its host. These do not replace the
VS Code extension's telemetry.telemetryLevel setting. See
Plugin telemetry details.
AI authoring, review and execution can send selected inputs to the configured
provider under their separate permissions. Saved run artifacts remain local
and can contain those inputs and outputs.
See VS Code telemetry documentation,
the GitHub Privacy Statement
and Microsoft Privacy Statement.
License
Licensed under the GitHub License Terms for Extensions. See the LICENSE
file included with the extension.