🧪 SS - Reqnroll Test Runner (VS Code extension)
Runs Reqnroll / SpecFlow .feature scenarios from VS Code's Testing panel, alongside C# Dev Kit rather than instead of it. Its tree, panel, settings and commands are all named SS - Reqnroll Tests, so they're easy to tell apart from Dev Kit's own ".NET" tests.
Zero-config. The test tree is built straight from the .feature files, so it appears as soon as the project opens — no build or discovery pass needed. The extension also auto-detects your test project, built assembly, features folder, root namespace, and every *.runsettings in the repo by scanning the workspace, so it works for any Reqnroll / SpecFlow (or other dotnet test) project with no settings at all. The cpTests.* settings remain as optional overrides.
Getting started
- Install the extension (see Install below) and open a workspace that contains
.feature files. There's nothing to configure.
- Open the Testing panel (beaker icon) — the 🧪 SS - Reqnroll Tests tree appears, built from your
.feature files.
- Pick an environment: click
$(beaker) Tests: … in the status bar and choose a *.runsettings file (all are auto-discovered).
- Run a scenario from the tree, the editor gutter ▶, or the 🚀 SS - Reqnroll Run Panel (activity-bar icon) where you can search and tick several.
- Debug with the 🐞 profile (your breakpoints are hit), or ⏸ Debug + Pause on failure to stop at the failing step with the browser open.
- When it finishes, click 📄 Report for the HTML report.
Features
- Instant, zero-config test tree. Organised as folders → features → scenarios → Scenario Outline example rows, built straight from the
.feature files and updated as soon as you save one. Project paths are auto-detected — no setup needed. Clicking a node opens the scenario line.
- Run and debug at any level. Run or debug a folder, a feature, a scenario or a single example row — from the Testing tree, the editor gutter ▶, or the Run Panel.
- Multi-select Run Panel. Search scenarios and tick several with checkboxes, then Run / Debug them together. The number selected shows as a blue badge.
- ⏸ Debug + Pause on failure. Debug a scenario and automatically pause at the first failing step with the browser kept open, the failing line highlighted, and a popup showing the error, the failing Gherkin step (click to open the
.feature line) and your stack frames (click to open the source). See below.
- Live results. Scenarios are marked Passed, Failed, Skipped, Cancelled or Error as each one finishes.
- Environment picker. The status bar shows
$(beaker) Tests: perseus_env. Click it to choose a *.runsettings file (auto-discovered across the repo) or to run without one. The choice is remembered per workspace. No confirmation prompt by default — opt in via cpTests.confirmEnvironments (e.g. ["prod"]).
- Builds only when needed. Every editor is saved before a run.
dotnet build then runs only if a source file changed since the last successful build (see How a run works). Referenced projects (via <ProjectReference>) are included automatically.
- One-click HTML report. A coloured 📄 Report button (green when passed, red when failed) on the run card and in history opens the newest HTML report in your
reportFolder; Open Last Report is in the Command Palette too.
- Cancel safely. Cancelling kills the whole
dotnet/testhost process tree, including the Playwright browsers.
- Run Panel. A form in the Testing sidebar for searching, multi-selecting and running tests, with live progress and run history (see below).
🚀 Reqnroll Run Panel
It sits in its own activity-bar container (SS - Reqnroll Tests) and under the test tree. You can also open it with 🧪 SS - Reqnroll Tests: Open Run Panel.
Layout adapts to the panel's width. In the sidebar everything is one column. Below ~260px each label sits above its field so inputs get the full width, and in the sidebar each recent run's buttons drop below its name. Drag the panel into the wide bottom area (≥640px) and Recent runs move beside the setup and current run, so a short, wide panel needs no scrolling.
Selecting what to run
| Control |
What it does |
| Runsettings file |
A path, as you'd type it after --settings on the command line, e.g. CP.Automation.Tests\RunSettings\perseus_env.runsettings. Every *.runsettings in the repo is suggested as you type (or just the runSettingsFolder if you set one). If it's empty, missing, or not a .runsettings file, the run goes without --settings and the panel says so. It's the same choice as the status bar |
| Scenario search |
Type part of a scenario name, feature name, or an Examples value (e.g. a username), then tick one or more results. A Scenario Outline lists its Examples rows as a tree under it: tick the scenario box to select every example, or tick/untick single examples (the scenario box shows – when only some are chosen, and the sub-line shows "2 of 9 examples"). Ticking every example one by one collapses back to the whole scenario. The dropdown stays open while you pick; the blue badge shows e.g. 1 scenario + 2 examples selected. Clear unticks all. The panel shows the exact --filter it will pass |
The ticked scenarios are combined into one --filter. With nothing ticked, the run covers everything (after a confirmation).
Then press ▶ Run (blue), 🐞 Debug (orange, the theme's debugging colour) or ⏸ Debug + Pause on failure (teal). Both debug colours come from VS Code theme tokens paired with their own text colour, so they stay readable in light, dark and high-contrast themes. Running several ticked scenarios executes them in parallel in one dotnet test; debugging several runs them one at a time, each in its own debug session.
Current run and Recent runs
- Current run is a live card showing a coloured status mark and word (Building, Running, Debugging, Passed ✔, Failed ✖, Skipped ⊘, Error ⚠, Cancelled), the scenario name, environment and elapsed time.
- The scenario name is a link — click it to open the scenario in its
.feature file.
- Each example/variant is listed with its status and duration; a failure shows its error message.
- When the run ends, a 📄 Open Report button appears — green if it passed, red if it failed.
- Recent runs keeps the last 10 runs, from both the panel and the Testing tab.
- Each shows the outcome word and a coloured mark, and the scenario name is clickable (opens the
.feature file).
- 📄 Report opens that run's HTML report (green/red by outcome).
- Panel runs also have Re-run, which repeats the same selection on the current environment.
test/liveCategories.test.ts and test/liveFilterExecution.test.ts (opt-in with CP_LIVE=1) check that the panel's count equals what vstest selects, and that every filter shape executes exactly the selected tests.
⏸ Debug + Pause on failure
A debug mode for working out why a UI step failed, without re-running the whole scenario. Start it from the Run Panel's ⏸ Debug + Pause on failure button, or the editor gutter ▶ → Execute Using Profile… → ⏸ Debug + Pause on failure.
When a step fails, the run pauses right there (the Playwright browser stays open so you can inspect the page) and the editor:
- Highlights the failing line in your page/step
.cs file (yellow line + gutter marker) and opens a popup (it stays until you press Escape, and scrolls) showing:
- Your code (click to open) — the failing Gherkin step (click → opens and highlights the matching line in the
.feature file) followed by each of your stack frames (click → opens that .cs line).
- The
ERROR : … message and Playwright call log, in red.
- Writes a partial HTML report up to the failure, reachable from the paused run card.
- Leaves a "Show test failure" CodeLens on the line so you can reopen the popup.
Press Continue (F5) to let the scenario finish failing, then fix the code and re-run. It only pauses when a debugger is attached and this mode was chosen — normal runs, plain Debug and CI are unaffected.
This needs a small hook in the test project's Hooks.cs (AfterStep) that writes the failure to a cp-pause.txt next to the test assembly and calls Debugger.Break(). Without it, Debug + Pause behaves like a normal Debug run.
Install
npm ci
npm run package # typecheck + unit tests + bundle + ss-reqnroll-test-runner-<version>.vsix
code --install-extension ss-reqnroll-test-runner-<version>.vsix
Alternatively, open the Extensions view, click ⋯, choose Install from VSIX… and pick the .vsix file. Reload the window afterwards.
Running next to C# Dev Kit
C# Dev Kit keeps working on its own, with its own ".NET" test tree, Solution Explorer and IntelliSense. This extension doesn't change or disable any of it. Both trees appear in the Testing panel: 🧪 SS - Reqnroll Tests is this one. You can hide either tree with the Testing view's filter menu. Debugging from this extension needs the C# extension (ms-dotnettools.csharp), which Dev Kit installs. (Our debug attach turns Hot Reload off for the test run, since it isn't useful there.)
How a run works
- Save all editors, then build your
cpTests.testProject only if needed.
- MSBuild can recompile generated
*.feature.cs on every build even when nothing changed, which is slow.
- Instead, the extension fingerprints the real inputs: the path, size and modified time of
.cs, .feature, .csproj, .json and similar files under cpTests.buildInputs, leaving out bin, obj, TestResults and the generated *.feature.cs. It skips the build when the fingerprint and the built DLL are both unchanged since the last successful build.
cpTests.noBuild never builds.
dotnet test <built dll> --settings <env>.runsettings --filter … --logger console;verbosity=normal --logger trx.
- Passing the DLL instead of the
.csproj keeps MSBuild from mangling the quotes and commas in the filter.
- Results are read live from the console output. Once the run ends, the TRX file fills in anything the console couldn't attribute, such as two features that share a scenario method name.
- The ExtentReports HTML that
Hooks.cs writes is picked up and offered through Open Report.
How names are matched
The extension works out NUnit test names exactly the way Reqnroll generates them:
| Part |
Rule |
Example |
| Namespace |
Root namespace plus each folder as an identifier |
MyProject.Tests.Features.Login |
| Class |
Feature title as an identifier, plus Feature |
LoginResetPasswordFeature |
| Method |
Scenario title as an identifier |
3a. Verify… becomes _3A_Verify… |
| Outline rows |
Rows are told apart by Reqnroll's @__pickleIndex argument |
|
test/generatedCorpus.test.ts checks these rules against every generated *.feature.cs on your machine. Those files are gitignored, so this test only runs after a local build.
If a test is reported as "No result was reported for this test (expected …)", the computed name didn't match anything in the DLL. Usually that means the DLL is stale (rebuild, or turn off cpTests.noBuild). Otherwise, run the corpus test to find the naming rule that's wrong.
Settings
All settings are optional — leave them blank and the extension auto-detects everything. Set one only to override the detection.
| Setting |
Auto-detected from (when blank) |
Override example |
cpTests.testProject |
the .csproj nearest the .feature files that references Reqnroll/SpecFlow |
MyProject.Tests/MyProject.Tests.csproj |
cpTests.testAssembly |
the built <Name>.dll under bin (or the csproj's target framework) |
MyProject.Tests/bin/Debug/net8.0/MyProject.Tests.dll |
cpTests.featuresRoot |
the common folder of the .feature files |
MyProject.Tests/Features |
cpTests.rootNamespace |
<RootNamespace> / <AssemblyName> of the test project |
MyProject.Tests |
cpTests.runSettingsFolder |
blank = suggest every *.runsettings in the repo |
MyProject.Tests/RunSettings |
cpTests.buildInputs |
the test project plus its <ProjectReference> graph |
["MyProject.Tests", "MyProject.Framework"] |
cpTests.reportFolder |
default TestResults |
TestResults |
cpTests.confirmEnvironments |
default [] (no prompt) |
["prod"] |
cpTests.noBuild |
default false |
true |
cpTests.extraArgs |
default [] |
["--blame"] |
For most projects you can delete the whole cpTests.* block from .vscode/settings.json and rely on auto-detection.
Development
npm test # unit tests + generated-code corpus check
$env:CP_LIVE=1; npx vitest run test/live*.test.ts # vs the built DLL (list-only) and the fixture project (runs fixture tests only)
npm run watch # rebuild dist/ on change; press F5 from this folder to debug the extension
Manual smoke checklist (after changes)
- [ ] Run one plain scenario, one example row and one whole feature. Each one shows live ticks and an Open Report button.
- [ ] Cancel a run partway through. No
dotnet, testhost or browser processes are left running.
- [ ] Debug a scenario with a breakpoint in a step definition. The breakpoint is hit.
- [ ] Debug + Pause on failure a scenario that fails. It pauses at the failing step with the browser open; the failing line is highlighted and the popup shows the error + clickable step/frames.
- [ ] Introduce a build error. Every queued test shows "Build failed" and the output opens.
- [ ] Choose
prod_env.runsettings and click Run. It runs with no prompt (add "cpTests.confirmEnvironments": ["prod"] to get a confirmation dialog).
- [ ] Type a path that doesn't exist. The panel warns, and the run's command has no
--settings.
- [ ] Run the same scenario twice without editing anything. The second run logs "No source changes … skipping dotnet build".
- [ ] In the Run Panel, search and tick two scenarios, then Run. Both run, and the blue 2 scenarios selected badge shows.
- [ ] Search an outline, untick one example under it, then Run. Only the remaining examples run; the box shows – and "N of M examples".
- [ ] Click the ▶ in the gutter of a single Examples row. Only that example runs; the ▶ on the
Scenario Outline: line runs all of them.
- [ ] Change the runsettings in the panel. The status bar follows, and the reverse works too.