CMake Test Explorer for Visual Studio Code

Run your CMake tests using Visual Studio Code's native Testing UI.

What's new in version 1.1.0

Added

• Bring back cmakeExplorer.buildDir setting. This fixes issue #78.
• Add cmakeExplorer.autodetectBuildDir setting. This fixes issue #80 along with the above.

Fixed

• Remove false positives in build dir detection. This fixes issue #79.

Features

• Autodects CMake build directories in workspace
• Reasonable defaults for zero-config
• Integrates with CMake Tools
• Runs tests in parallel
• Supports custom test properties for advanced features like hierarchical suites and test locations

Configuration

Variable substitution

Some options support the replacement of special values in their string value by using a ${variable} syntax.

The following built-in variables are expanded:

Environments variables are prefixed with env:. For example ${env:HOME} will be substituted with the home path on Unix systems.

(Note: On Windows, variable names are case insensitive but must be uppercase for env: substitition to work properly)

Additionally, if the CMake Tools extension is active in the current workspace and cmakeExplorer.cmakeIntegration is enabled, then the following variables can be used:

If you want the extension to infer the right configuration automatically from CMake Tools, simply keep these default settings:

If these variables are missing from the current settings (for example, if the CMake Tools extension is missing or disabled) then they are substituted with an empty string, falling back to default behavior.

Note that any change to the CMake Tools configuration, either from the settings or the status bar, requires a manual test reload from the Testing UI.

Source files

The Testing UI has a feature to link tests with their source files. CMake provides the set_tests_properties() command to associate tests with various metadata, however it only support a predefined list of properties, and none of them seems suitable for this purpose. To support this feature anyway, the extension expects that the file path and line number be passed as test environment variables using the ENVIRONMENT property, like so:

add_test(
    NAME <name>
    COMMAND <command> [<args> ...]
)
set_tests_properties(<name> PROPERTIES
    ENVIRONMENT "TEST_FILE=<filename>;TEST_LINE=<line>"
)

Here we are using TEST_FILE and TEST_LINE environment variables but you are free to choose other variable names. You can then edit the cmakeExplorer.testFileVar and cmakeExplorer.testLineVar settings accordingly, and you should see an extra 'Go to Test' entry in the contextual menu.

Note that the cmakeExplorer.testFileVar setting must be set for these features to work, however if the cmakeExplorer.testLineVar setting is missing or the variable is not set for a given test then the 'Go to Test' will simply open the file.

Error pattern

The cmakeExplorer.errorPattern setting can be used to capture error messages on the test output. If the testExplorer.useNativeTesting setting is enabled, the captured messages will be displayed in the file editor next to the line where the error occurred.

This setting expects a regular expression string with the following named capturing groups:

• file
• line
• severity (optional)
• message

See the MDN documentation on regular expression groups for more information on the syntax. For example, here is the default value that captures GCC-style error messages:

^(?<file>[^<].*?):(?<line>\\d+):\\d*:?\\s+(?<severity>(?:fatal\\s+)?(?:warning|error)):\\s+(?<message>.*)$

This pattern will match the following error message:

/path/to/my/file.c:123: error: unexpected value

The captured groups will be:

The advantage of using named groups is that their order does not matter. Let's consider the following error message:

ERROR: assertion failed on line 123 (file=/path/to/my/file.c)

To capture this message you could use the following syntax:

^(?<severity>ERROR):\s+(?<message>.*)\s+on line (?<line>\d+) \(file=(?<file>.+)\)$

Debugging

The extension comes pre-configured with sensible defaults for debugging tests:

{
	"name": "CTest",
	"type": "cppdbg",
	"request": "launch",
	"windows": {
		"type": "cppvsdbg"
	},
	"linux": {
		"type": "cppdbg",
		"MIMode": "gdb"
	},
	"osx": {
		"type": "cppdbg",
		"MIMode": "lldb"
	}
}

You can also use a custom configuration defined in the standard launch.json. To do so, edit the cmakeExplorer.debugConfig setting with the name of the debug configuration to use.

Debugging a test will overwrite the following debug configuration fields with values from the CTest metadata:

For example, if you want the debugger to stop at the entry point of your tests, add the following config in your launch.json then set cmakeExplorer.debugConfig to "myCustomDebugConfig" :

{
	"name": "myCustomDebugConfig",
	"type": "cppdbg",
	"request": "launch",
	"stopAtEntry": true,
	"windows": {
		"type": "cppvsdbg"
	}
}

Parallel test jobs

The extension can run test jobs in parallel. The maximum number of jobs to run is the first non-zero value in the following order:

• The cmakeExplorer.parallelJobs setting (see Configuration)
• The cmake.ctest.parallelJobs then cmake.parallelJobs settings if the CMake Tools extension is installed
• The number of processors on the machine

A negative value will disable parallel execution.

The extension uses the native CTest parallel execution support (i.e. the -j|--parallel command-line option).

Troubleshooting

First, make sure that CTest works from the command line. Some issues come from the CTest configuration and not the extension itself. See issues #10 and #14 for examples of such cases.

The Testing UI displays an error

Clicking on the error message in the Testing panel should open the log panel with the output of the CTest command used by the extension to load the test list.

• SyntaxError: Unexpected token T in JSON at position 0

  The extension requires CTest option --show-only=json-v1 to load the test list. This option was introduced with CMake version 3.14. Make sure to use a version that supports this flag. See issue #2.

• Error: CMake cache file /path/to/project/${buildDirectory}/CMakeCache.txt does not exist

  The cmakeExplorer.cmakeIntegration flag is enabled by default. This adds support for extra variables in other settings (See Variable substitution for more info). If the extension is not installed or active then these variables are not substituted. You can activate the extension's log panel in the settings for more details.

The Testing UI shows no error but the test list is empty

The extension autodetects CMake build directories with tests enabled by finding all directories in the workspace having both CMakeCache.txt and CTestTestfile.cmake. Ensure you have at least one and run ctest -C Debug from there.