Skip to content
| Marketplace
Sign in
Visual Studio Code>Testing>🧪 SS - Reqnroll Test RunnerNew to Visual Studio Code? Get it now.
🧪 SS - Reqnroll Test Runner

🧪 SS - Reqnroll Test Runner

Varun Kumar Gopinath

|
25 installs
| (0) | Free
Zero-config runner for Reqnroll / SpecFlow .feature scenarios in VS Code's Testing panel — auto-detects your test project, assembly, features and runsettings. Multi-select run panel, live results, one-click HTML reports, and a Debug + Pause-on-failure mode that stops at the failing step with the bro
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

🧪 SS - Reqnroll Test Runner (VS Code extension)

Runs Reqnroll / SpecFlow .feature scenarios from VS Code's Testing panel, alongside C# Dev Kit rather than instead of it. Its tree, panel, settings and commands are all named SS - Reqnroll Tests, so they're easy to tell apart from Dev Kit's own ".NET" tests.

Zero-config. The test tree is built straight from the .feature files, so it appears as soon as the project opens — no build or discovery pass needed. The extension also auto-detects your test project, built assembly, features folder, root namespace, and every *.runsettings in the repo by scanning the workspace, so it works for any Reqnroll / SpecFlow (or other dotnet test) project with no settings at all. The cpTests.* settings remain as optional overrides.

Getting started

  1. Install the extension (see Install below) and open a workspace that contains .feature files. There's nothing to configure.
  2. Open the Testing panel (beaker icon) — the 🧪 SS - Reqnroll Tests tree appears, built from your .feature files.
  3. Pick an environment: click $(beaker) Tests: … in the status bar and choose a *.runsettings file (all are auto-discovered).
  4. Run a scenario from the tree, the editor gutter ▶, or the 🚀 SS - Reqnroll Run Panel (activity-bar icon) where you can search and tick several.
  5. Debug with the 🐞 profile (your breakpoints are hit), or ⏸ Debug + Pause on failure to stop at the failing step with the browser open.
  6. When it finishes, click 📄 Report for the HTML report.

Features

  • Instant, zero-config test tree. Organised as folders → features → scenarios → Scenario Outline example rows, built straight from the .feature files and updated as soon as you save one. Project paths are auto-detected — no setup needed. Clicking a node opens the scenario line.
  • Run and debug at any level. Run or debug a folder, a feature, a scenario or a single example row — from the Testing tree, the editor gutter ▶, or the Run Panel.
  • Multi-select Run Panel. Search scenarios and tick several with checkboxes, then Run / Debug them together. The number selected shows as a blue badge.
  • ⏸ Debug + Pause on failure. Debug a scenario and automatically pause at the first failing step with the browser kept open, the failing line highlighted, and a popup showing the error, the failing Gherkin step (click to open the .feature line) and your stack frames (click to open the source). See below.
  • Live results. Scenarios are marked Passed, Failed, Skipped, Cancelled or Error as each one finishes.
  • Environment picker. The status bar shows $(beaker) Tests: perseus_env. Click it to choose a *.runsettings file (auto-discovered across the repo) or to run without one. The choice is remembered per workspace. No confirmation prompt by default — opt in via cpTests.confirmEnvironments (e.g. ["prod"]).
  • Builds only when needed. Every editor is saved before a run. dotnet build then runs only if a source file changed since the last successful build (see How a run works). Referenced projects (via <ProjectReference>) are included automatically.
  • One-click HTML report. A coloured 📄 Report button (green when passed, red when failed) on the run card and in history opens the newest HTML report in your reportFolder; Open Last Report is in the Command Palette too.
  • Cancel safely. Cancelling kills the whole dotnet/testhost process tree, including the Playwright browsers.
  • Run Panel. A form in the Testing sidebar for searching, multi-selecting and running tests, with live progress and run history (see below).

🚀 Reqnroll Run Panel

It sits in its own activity-bar container (SS - Reqnroll Tests) and under the test tree. You can also open it with 🧪 SS - Reqnroll Tests: Open Run Panel.

Layout adapts to the panel's width. In the sidebar everything is one column. Below ~260px each label sits above its field so inputs get the full width, and in the sidebar each recent run's buttons drop below its name. Drag the panel into the wide bottom area (≥640px) and Recent runs move beside the setup and current run, so a short, wide panel needs no scrolling.

Selecting what to run

Control What it does
Runsettings file A path, as you'd type it after --settings on the command line, e.g. CP.Automation.Tests\RunSettings\perseus_env.runsettings. Every *.runsettings in the repo is suggested as you type (or just the runSettingsFolder if you set one). If it's empty, missing, or not a .runsettings file, the run goes without --settings and the panel says so. It's the same choice as the status bar
Scenario search Type part of a scenario name, feature name, or an Examples value (e.g. a username), then tick one or more results. A Scenario Outline lists its Examples rows as a tree under it: tick the scenario box to select every example, or tick/untick single examples (the scenario box shows – when only some are chosen, and the sub-line shows "2 of 9 examples"). Ticking every example one by one collapses back to the whole scenario. The dropdown stays open while you pick; the blue badge shows e.g. 1 scenario + 2 examples selected. Clear unticks all. The panel shows the exact --filter it will pass

The ticked scenarios are combined into one --filter. With nothing ticked, the run covers everything (after a confirmation).

Then press ▶ Run (blue), 🐞 Debug (orange, the theme's debugging colour) or ⏸ Debug + Pause on failure (teal). Both debug colours come from VS Code theme tokens paired with their own text colour, so they stay readable in light, dark and high-contrast themes. Running several ticked scenarios executes them in parallel in one dotnet test; debugging several runs them one at a time, each in its own debug session.

Current run and Recent runs

  • Current run is a live card showing a coloured status mark and word (Building, Running, Debugging, Passed ✔, Failed ✖, Skipped ⊘, Error ⚠, Cancelled), the scenario name, environment and elapsed time.
    • The scenario name is a link — click it to open the scenario in its .feature file.
    • Each example/variant is listed with its status and duration; a failure shows its error message.
    • When the run ends, a 📄 Open Report button appears — green if it passed, red if it failed.
  • Recent runs keeps the last 10 runs, from both the panel and the Testing tab.
    • Each shows the outcome word and a coloured mark, and the scenario name is clickable (opens the .feature file).
    • 📄 Report opens that run's HTML report (green/red by outcome).
    • Panel runs also have Re-run, which repeats the same selection on the current environment.

test/liveCategories.test.ts and test/liveFilterExecution.test.ts (opt-in with CP_LIVE=1) check that the panel's count equals what vstest selects, and that every filter shape executes exactly the selected tests.

⏸ Debug + Pause on failure

A debug mode for working out why a UI step failed, without re-running the whole scenario. Start it from the Run Panel's ⏸ Debug + Pause on failure button, or the editor gutter ▶ → Execute Using Profile… → ⏸ Debug + Pause on failure.

When a step fails, the run pauses right there (the Playwright browser stays open so you can inspect the page) and the editor:

  • Highlights the failing line in your page/step .cs file (yellow line + gutter marker) and opens a popup (it stays until you press Escape, and scrolls) showing:
    • Your code (click to open) — the failing Gherkin step (click → opens and highlights the matching line in the .feature file) followed by each of your stack frames (click → opens that .cs line).
    • The ERROR : … message and Playwright call log, in red.
  • Writes a partial HTML report up to the failure, reachable from the paused run card.
  • Leaves a "Show test failure" CodeLens on the line so you can reopen the popup.

Press Continue (F5) to let the scenario finish failing, then fix the code and re-run. It only pauses when a debugger is attached and this mode was chosen — normal runs, plain Debug and CI are unaffected.

This needs a small hook in the test project's Hooks.cs (AfterStep) that writes the failure to a cp-pause.txt next to the test assembly and calls Debugger.Break(). Without it, Debug + Pause behaves like a normal Debug run.

Install

npm ci
npm run package          # typecheck + unit tests + bundle + ss-reqnroll-test-runner-<version>.vsix
code --install-extension ss-reqnroll-test-runner-<version>.vsix

Alternatively, open the Extensions view, click ⋯, choose Install from VSIX… and pick the .vsix file. Reload the window afterwards.

Running next to C# Dev Kit

C# Dev Kit keeps working on its own, with its own ".NET" test tree, Solution Explorer and IntelliSense. This extension doesn't change or disable any of it. Both trees appear in the Testing panel: 🧪 SS - Reqnroll Tests is this one. You can hide either tree with the Testing view's filter menu. Debugging from this extension needs the C# extension (ms-dotnettools.csharp), which Dev Kit installs. (Our debug attach turns Hot Reload off for the test run, since it isn't useful there.)

How a run works

  1. Save all editors, then build your cpTests.testProject only if needed.
    • MSBuild can recompile generated *.feature.cs on every build even when nothing changed, which is slow.
    • Instead, the extension fingerprints the real inputs: the path, size and modified time of .cs, .feature, .csproj, .json and similar files under cpTests.buildInputs, leaving out bin, obj, TestResults and the generated *.feature.cs. It skips the build when the fingerprint and the built DLL are both unchanged since the last successful build.
    • cpTests.noBuild never builds.
  2. dotnet test <built dll> --settings <env>.runsettings --filter … --logger console;verbosity=normal --logger trx.
    • Passing the DLL instead of the .csproj keeps MSBuild from mangling the quotes and commas in the filter.
  3. Results are read live from the console output. Once the run ends, the TRX file fills in anything the console couldn't attribute, such as two features that share a scenario method name.
  4. The ExtentReports HTML that Hooks.cs writes is picked up and offered through Open Report.

How names are matched

The extension works out NUnit test names exactly the way Reqnroll generates them:

Part Rule Example
Namespace Root namespace plus each folder as an identifier MyProject.Tests.Features.Login
Class Feature title as an identifier, plus Feature LoginResetPasswordFeature
Method Scenario title as an identifier 3a. Verify… becomes _3A_Verify…
Outline rows Rows are told apart by Reqnroll's @__pickleIndex argument

test/generatedCorpus.test.ts checks these rules against every generated *.feature.cs on your machine. Those files are gitignored, so this test only runs after a local build.

If a test is reported as "No result was reported for this test (expected …)", the computed name didn't match anything in the DLL. Usually that means the DLL is stale (rebuild, or turn off cpTests.noBuild). Otherwise, run the corpus test to find the naming rule that's wrong.

Settings

All settings are optional — leave them blank and the extension auto-detects everything. Set one only to override the detection.

Setting Auto-detected from (when blank) Override example
cpTests.testProject the .csproj nearest the .feature files that references Reqnroll/SpecFlow MyProject.Tests/MyProject.Tests.csproj
cpTests.testAssembly the built <Name>.dll under bin (or the csproj's target framework) MyProject.Tests/bin/Debug/net8.0/MyProject.Tests.dll
cpTests.featuresRoot the common folder of the .feature files MyProject.Tests/Features
cpTests.rootNamespace <RootNamespace> / <AssemblyName> of the test project MyProject.Tests
cpTests.runSettingsFolder blank = suggest every *.runsettings in the repo MyProject.Tests/RunSettings
cpTests.buildInputs the test project plus its <ProjectReference> graph ["MyProject.Tests", "MyProject.Framework"]
cpTests.reportFolder default TestResults TestResults
cpTests.confirmEnvironments default [] (no prompt) ["prod"]
cpTests.noBuild default false true
cpTests.extraArgs default [] ["--blame"]

For most projects you can delete the whole cpTests.* block from .vscode/settings.json and rely on auto-detection.

Development

npm test                                  # unit tests + generated-code corpus check
$env:CP_LIVE=1; npx vitest run test/live*.test.ts   # vs the built DLL (list-only) and the fixture project (runs fixture tests only)
npm run watch                             # rebuild dist/ on change; press F5 from this folder to debug the extension

Manual smoke checklist (after changes)

  • [ ] Run one plain scenario, one example row and one whole feature. Each one shows live ticks and an Open Report button.
  • [ ] Cancel a run partway through. No dotnet, testhost or browser processes are left running.
  • [ ] Debug a scenario with a breakpoint in a step definition. The breakpoint is hit.
  • [ ] Debug + Pause on failure a scenario that fails. It pauses at the failing step with the browser open; the failing line is highlighted and the popup shows the error + clickable step/frames.
  • [ ] Introduce a build error. Every queued test shows "Build failed" and the output opens.
  • [ ] Choose prod_env.runsettings and click Run. It runs with no prompt (add "cpTests.confirmEnvironments": ["prod"] to get a confirmation dialog).
  • [ ] Type a path that doesn't exist. The panel warns, and the run's command has no --settings.
  • [ ] Run the same scenario twice without editing anything. The second run logs "No source changes … skipping dotnet build".
  • [ ] In the Run Panel, search and tick two scenarios, then Run. Both run, and the blue 2 scenarios selected badge shows.
  • [ ] Search an outline, untick one example under it, then Run. Only the remaining examples run; the box shows – and "N of M examples".
  • [ ] Click the ▶ in the gutter of a single Examples row. Only that example runs; the ▶ on the Scenario Outline: line runs all of them.
  • [ ] Change the runsettings in the panel. The status bar follows, and the reverse works too.
  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
  • Your Privacy Choices
  • Consumer Health Privacy
© 2026 Microsoft