🧪 Reqnroll Test Runner (VS Code extension)

Runs Reqnroll / SpecFlow .feature scenarios from VS Code's Testing panel, alongside C# Dev Kit rather than instead of it. Its tree, panel, settings and commands are all named Reqnroll Tests, so they're easy to tell apart from Dev Kit's own ".NET" tests.

The extension builds the test tree directly from the .feature files, so it appears as soon as the project opens — no build or discovery pass needed. Point it at your test project with the cpTests.* settings and it works for any Reqnroll / SpecFlow (or other dotnet test) project.

Features

Instant test tree. The tree is organised as folders, then features, then scenarios, then Scenario Outline example rows. It updates as soon as you save a .feature file. Clicking a node opens the scenario line.
Run and debug at any level. You can run or debug a folder, a feature, a scenario or a single example row.
Live results. Scenarios are marked passed, failed or skipped as each one finishes. Failures show the error message and stack trace inline.
Runsettings picker. The status bar shows $(beaker) CP: staging_env. Click it to choose a *.runsettings file, or to run without one. The choice is remembered per workspace, and runsettings whose file name contains prod ask for confirmation before every run.
Builds only when needed. Every editor is saved before a run. dotnet build then runs only if a source file has changed since the last successful build (see How a run works).
Open Report. When a run finishes, a notification shows the counts and an Open Report button. The button opens the newest HTML report in your reportFolder in your browser. You can also run 🧪 Reqnroll Tests: Open Last Report from the Command Palette later.
Cancel. Cancelling a run kills the whole dotnet process tree, including the Playwright browsers.
Run Panel. A form in the Testing sidebar, below the test tree, for searching, filtering and running tests, with live progress and run history (see below).

🚀 Reqnroll Run Panel

It sits under the test tree in the Testing sidebar. You can also open it with 🧪 Reqnroll Tests: Open Run Panel.

Run setup

Control	What it does
Runsettings file	A path, as you'd type it after `--settings` on the command line, e.g. `Tests\RunSettings\staging_env.runsettings`. The files in your `runSettingsFolder` are suggested as you type. If it's empty, the file doesn't exist, or it isn't a `.runsettings` file, the run goes without `--settings` and the panel says so. It's the same setting as the status bar
Search scenarios	Type part of a scenario or feature name. Tick the results you want to run, or click a title to open the scenario in its `.feature` file
Tag (left)	One scenario or example tag, picked from a searchable dropdown of every tag in the repo
Filter (right)	Any `dotnet test --filter` expression, e.g. `TestCategory=cch\\|TestCategory=go` or `FullyQualifiedName~SSR`

The ticked scenarios, the tag and the filter are combined with AND. The panel shows the exact --filter it will pass.

Test count: when the filter only uses TestCategory terms (e.g. TestCategory=nick or TestCategory=gathersettings&TestCategory!=cch), the count comes from the .feature index and is exact, and the tests are queued in the Testing tree. For other filters the count is unknown until the run starts, and tests appear in the tree as their results arrive.

Current run and Recent runs

Current run is a live card with the status (Building, Running, Passed, Failed, Cancelled), environment, elapsed time, a progress bar and the passed/failed/skipped counts.
- Each test is listed with its status and duration. Failures come first, each with its error message.
- Click a test to open its scenario.
- When the run ends, a green 📄 Open Report button appears.
Recent runs keeps the last 10 runs, from both the panel and the Testing tab.
- Each run has a green 📄 Report button, which opens that run's HTML report.
- Panel runs also have Re-run, which repeats the same selection on the current environment.

test/liveCategories.test.ts and test/liveFilterExecution.test.ts (opt-in with CP_LIVE=1) check that the panel's count equals what vstest selects, and that every filter shape executes exactly the selected tests.

Install

npm ci
npm run package          # typecheck + unit tests + bundle + reqnroll-test-runner-<version>.vsix
code --install-extension reqnroll-test-runner-<version>.vsix

Alternatively, open the Extensions view, click ⋯, choose Install from VSIX… and pick the .vsix file. Reload the window afterwards.

Running next to C# Dev Kit

C# Dev Kit keeps working on its own, with its own ".NET" test tree, Solution Explorer, IntelliSense and Hot Reload. This extension doesn't change or disable any of it. Both trees appear in the Testing panel: 🧪 Reqnroll Tests is this one. You can hide either tree with the Testing view's filter menu. Debugging from this extension needs the C# extension (ms-dotnettools.csharp), which Dev Kit installs.

How a run works

Save all editors, then build your cpTests.testProject only if needed.
- MSBuild can recompile generated *.feature.cs on every build even when nothing changed, which is slow.
- Instead, the extension fingerprints the real inputs: the path, size and modified time of .cs, .feature, .csproj, .json and similar files under cpTests.buildInputs, leaving out bin, obj, TestResults and the generated *.feature.cs. It skips the build when the fingerprint and the built DLL are both unchanged since the last successful build.
- cpTests.noBuild never builds.
dotnet test <built dll> --settings <env>.runsettings --filter … --logger console;verbosity=normal --logger trx.
- Passing the DLL instead of the .csproj keeps MSBuild from mangling the quotes and commas in the filter.
Results are read live from the console output. Once the run ends, the TRX file fills in anything the console couldn't attribute, such as two features that share a scenario method name.
The ExtentReports HTML that Hooks.cs writes is picked up and offered through Open Report.

How names are matched

The extension works out NUnit test names exactly the way Reqnroll generates them:

Part	Rule	Example
Namespace	Root namespace plus each folder as an identifier	`MyProject.Tests.Features.Login`
Class	Feature title as an identifier, plus `Feature`	`LoginResetPasswordFeature`
Method	Scenario title as an identifier	`3a. Verify…` becomes `_3A_Verify…`
Outline rows	Rows are told apart by Reqnroll's `@__pickleIndex` argument

test/generatedCorpus.test.ts checks these rules against every generated *.feature.cs on your machine. Those files are gitignored, so this test only runs after a local build.

If a test is reported as "No result was reported for this test (expected …)", the computed name didn't match anything in the DLL. Usually that means the DLL is stale (rebuild, or turn off cpTests.noBuild). Otherwise, run the corpus test to find the naming rule that's wrong.

Settings

Setting	Example / Default
`cpTests.testProject`	`MyProject.Tests/MyProject.Tests.csproj`
`cpTests.testAssembly`	`MyProject.Tests/bin/Debug/net8.0/MyProject.Tests.dll`
`cpTests.featuresRoot`	`MyProject.Tests/Features`
`cpTests.rootNamespace`	`MyProject.Tests`
`cpTests.runSettingsFolder`	`MyProject.Tests/RunSettings`
`cpTests.reportFolder`	`TestResults` (default)
`cpTests.confirmEnvironments`	`["prod"]` (default)
`cpTests.noBuild`	`false` (default)
`cpTests.buildInputs`	`["MyProject.Tests"]`
`cpTests.extraArgs`	`[]` (default)

The path/namespace settings have no default — set them in your workspace .vscode/settings.json to match your project.

Development

npm test                                  # unit tests + generated-code corpus check
$env:CP_LIVE=1; npx vitest run test/live*.test.ts   # vs the built DLL (list-only) and the fixture project (runs fixture tests only)
npm run watch                             # rebuild dist/ on change; press F5 from this folder to debug the extension

Manual smoke checklist (after changes)

[ ] Run one plain scenario, one example row and one whole feature on staging. Each one shows live ticks and an Open Report button.
[ ] Cancel a run partway through. No dotnet, testhost or browser processes are left running.
[ ] Debug a scenario with a breakpoint in a step definition. The breakpoint is hit.
[ ] Introduce a build error. Every queued test shows "Build failed" and the output opens.
[ ] Choose prod_env.runsettings and click Run. A confirmation dialog appears.
[ ] Type a path that doesn't exist. The panel warns, and the run's command has no --settings.
[ ] Run the same scenario twice without editing anything. The second run logs "No source changes … skipping dotnet build".
[ ] In the Run Panel, search for a scenario, tick it and run it. Only that scenario runs, and the Current run card updates live.
[ ] Change the runsettings in the panel. The status bar follows, and the reverse works too.

🧪 SS - Reqnroll Test Runner

Varun Kumar Gopinath