CMake Test Explorer for Visual Studio Code

Run your CMake tests using the Test Explorer UI.

Features

• Shows a Test Explorer in the Test view in VS Code's sidebar with all detected tests and suites and their state
• Shows a failed test's log when the test is selected in the explorer
• Forwards the console output from the test executable to a VS Code output channel

Getting started

• Install the extension
• Open the Test view
• Run your tests using the icon in the Test Explorer

Configuration

Variable substitution

Some options support the replacement of special values in their string value by using a ${variable} syntax.

The following built-in variables are expanded:

Environments variables are prefixed with env:. For example ${env:HOME} will be substituted with the home path on Unix systems.

(Note: On Windows, variable names are case insensitive but must be uppercase for env: substitition to work properly)

Additionally, if the CMake Tools extension is active in the current workspace and cmakeExplorer.cmakeIntegration is enabled, then the following variables can be used:

If you want the Test Explorer to infer the right configuration automatically from CMake Tools, simply use these default settings:

If these variables are missing from the current settings (for example, if the CMake Tools extension is missing or disabled) then they are substituted with an empty string, falling back to default behavior.

Note that any change to the CMake Tools configuration, either from the settings or the status bar, requires a manual test reload from the Test Explorer sidebar.

Source files

The Test Explorer UI has a feature to link tests with their source files. CMake provides the set_tests_properties() command to associate tests with various metadata, however it only support a predefined list of properties, and none of them seems suitable for this purpose. To support this feature anyway, the extension expects that the file path and line number be passed as test environment variables using the ENVIRONMENT property, like so:

add_test(
    NAME <name>
    COMMAND <command> [<args> ...]
)
set_tests_properties(<name> PROPERTIES
    ENVIRONMENT "TEST_FILE=<filename>;TEST_LINE=<line>"
)

Here we are using TEST_FILE and TEST_LINE environment variables but you are free to choose other variable names. You can then edit the cmakeExplorer.testFileVar and cmakeExplorer.testLineVar settings accordingly, and you should see an extra 'Show source' icon appear in the Test Explorer panel next to all the tests where these variables are provided. This feature also enables extra Test Explorer UI such as editor decorations in the relevant source files (e.g. CodeLens, error messages etc).

Note that the cmakeExplorer.testFileVar setting must be set for these features to work, however if the cmakeExplorer.testLineVar setting is missing or the variable is not set for a given test then the 'Show source' will still work but the UI and commands provided by the core Test Explorer UI extension will work differently (e.g. 'Run tests in current file' instead of 'Run the test at the current cursor position', no CodeLens, etc). This can be useful when no line number information is available (shell scripts for example).

Error pattern

The cmakeExplorer.errorPattern setting can be used to capture error messages on the test output. If the testExplorer.useNativeTesting setting is enabled, the captured messages will be displayed in the file editor next to the line where the error occurred.

This setting expects a regular expression string with the following named capturing groups:

• file
• line
• severity (optional)
• message

See the MDN documentation on regular expression groups for more information on the syntax. For example, here is the default value that captures GCC-style error messages:

^(?<file>[^<].*?):(?<line>\\d+):\\d*:?\\s+(?<severity>(?:fatal\\s+)?(?:warning|error)):\\s+(?<message>.*)$

This pattern will match the following error message:

/path/to/my/file.c:123: error: unexpected value

The captured groups will be:

The advantage of using named groups is that their order does not matter. Let's consider the following error message:

ERROR: assertion failed on line 123 (file=/path/to/my/file.c)

To capture this message you could use the following syntax:

^(?<severity>ERROR):\s+(?<message>.*)\s+on line (?<line>\d+) \(file=(?<file>.+)\)$

Debugging

The extension comes pre-configured with sensible defaults for debugging tests:

{
  "name": "CTest",
  "type": "cppdbg",
  "request": "launch",
  "windows": {
    "type": "cppvsdbg"
  },
  "linux": {
    "type": "cppdbg",
    "MIMode": "gdb"
  },
  "osx": {
    "type": "cppdbg",
    "MIMode": "lldb"
  }
}

You can also use a custom configuration defined in the standard launch.json. To do so, edit the cmakeExplorer.debugConfig setting with the name of the debug configuration to use.

Debugging a test will overwrite the following debug configuration fields with values from the CTest metadata:

For example, if you want the debugger to stop at the entry point of your tests, add the following config in your launch.json then set cmakeExplorer.debugConfig to "myCustomDebugConfig" :

{
  "name": "myCustomDebugConfig",
  "type": "cppdbg",
  "request": "launch",
  "stopAtEntry": true,
  "windows": {
    "type": "cppvsdbg"
  }
}

Parallel test jobs

The extension can run test jobs in parallel. The maximum number of jobs to run is the first non-zero value in the following order:

• The cmakeExplorer.parallelJobs setting (see Configuration)
• The cmake.ctest.parallelJobs then cmake.parallelJobs settings if the CMake Tools extension is installed
• The number of processors on the machine

A negative value will disable parallel execution.

The extension uses the native CTest parallel execution support (i.e. the -j|--parallel command-line option).

Troubleshooting

First, make sure that CTest works from the command line. Some issues come from the CTest configuration and not the extension itself. See issues #10 and #14 for examples of such cases.

The Test Explorer panel displays an error

Clicking on the error message in the Test Explorer panel should open the log panel with the output of the CTest command used by the extension to load the test list.

• SyntaxError: Unexpected token T in JSON at position 0

  The extension requires CTest option --show-only=json-v1 to load the test list. This option was introduced with CMake version 3.14. Make sure to use a version that supports this flag. See issue #2.

• Error: CMake cache file /path/to/project/${buildDirectory}/CMakeCache.txt does not exist

  The cmakeExplorer.cmakeIntegration flag is enabled by default. This adds support for extra variables in other settings (See Variable substitution for more info). If the extension is not installed or active then these variables are not substituted. You can activate the extension's log panel in the settings for more details.

The Test Explorer panel shows no error but the test list is empty

Make sure that the cmakeExplorer.buildDir is properly configured. By default its value is empty, and in this case the extension shows no error if it fails to find the CMakeCache.txt file, in order not to clutter the Test Explorer panel for projects that don't use CMake.