Cantata Coverage Extension for Visual Studio Code

Cantata is a Unit and Integration test tool for C and C++.
This extension adds features into Visual Studio Code that allow Cantata coverage data to be overlaid on top of source code making it very quick and easy to see where further testing is required to meet a particular coverage level or goal.

Table of Contents

1. Features
2. Requirements
3. Provided Visual Studio Code Commands
4. Example usage without existing Cantata tests
5. Example usage for existing Cantata tests
6. Known issues
7. Release Notes

Features

• Easily enable Cantata coverage gathering for existing tests (or complete applications) even if they are not Cantata tests
• Set the Options to use for controlling Cantata coverage gathering
• View the gathered coverage as source code highlighting inside the C/C++ code editor
• Generate a Cantata coverage report

Requirements

A Cantata v22.10 or later installation with valid license is required for Cantata coverage instrumentation, generation of Cantata reports and display of gathered coverage data.
The extension will install, load and run without a licensed Cantata installation, however the above features will not all be available.
A free trial of Cantata can be obtained by visiting the QA Systems website.

Provided Visual Studio Code commands

This extension provides 4 new commands for use within the Command Palette all prefixed with 'Cantata':

• Cantata: Show Coverage
• Cantata: Hide Coverage
• Cantata: Edit Options File (UI)
• Cantata: Generate Coverage Report
• Cantata: Check Installation Information

Show/Hide Coverage

This command will cause Cantata coverage data to be overlaid on top of the code currently open C or C++ file, if such data exists.
If it doesn't exist, or it cannot be created, a notification is displayed indicating why.

The Coverage overlay replaces the syntax highlighting of the Editor with a coverage overlay that uses a simple traffic light colour system to indicate coverage achieved or not:

• Code coloured green
  • Indicates code that has been executed to the appropriate level for the coverage metrics used.
• Code coloured yellow/amber
  • Indicates code that has been partially executed.
  • For example the 'true' branch of a decision has been executed but the 'false' branch has not.
• Code coloured red
  • Indicates code that has not been executed.

For example:

Note that the coverage annotations will include individual operands/operators within each condition when using the logical coverage metric. This can be seen in the above screenshots.
When working with the Cantata Coverage Extension it is recommended that keyboard shortcuts are used to show/hide the coverage annotations.

Edit Options File (UI)

This command opens the UI for editing the options file used by the Cantata coverage instrumenter.
The options file itself can be edited using a simple text editor if desired. All settings available for use in the options file can be found in the main Cantata User Documentation.
The options file Editor provides some quick and simple controls for modifying the options file.

• Coverage Rule Set
  • This allows selection of a predefined set of coverage types, to quickly select the set of coverage metrics required by a particular safety standard.
• Alternatively, a set of checkboxes is provided allowing quick selection of the individual coverage metrics
  • Note that the Logical coverage metric is used for all boolean coverage metrics.
  • A full description of the coverage metrics can be found in the Cantata user Documentation.

Generate Coverage Report

Once Cantata coverage has been obtained for a source file, this command can be used to produce a Cantata Coverage report in the form of a CTR file suitable for certification purposes. This command calls the Cantata command line tool called CPPGETCOV to produce the report.
The generated CTR file is placed next to the selected source file.
See the main Cantata User Documentation for details about the CTR file content.

Check Installation Information

This command performs some basic checks relating to the Cantata installation on the current system.
It is executed upon startup of VSCode, and checks that:

• The Cantata executables can be found on the PATH environment variable

An information notification will be shown indicating where the Cantata installation was found, or a warning notification will be shown if the installation was not found.

Example Usage Without existing Cantata tests

This extension has been designed to make it simple to add Cantata Standalone Coverage to an existing application or test.
Prior to using the Cantata Extensions it is recommended the C/C++ extension from Microsoft is installed into Visual Studio Code if it is not already.

The basic steps to enable Cantata Coverage, and view the results within VSCode are as described below:

1. Create a new text file in the project in VSCode

• This is usually called ipg.cop for Cantata options files, but can be named anything.
• Add the following lines into the file:

#tool.use=true
--analyse
--comp:<deployment name>

Where <deployment name> should be replaced by the name of the Cantata deployment to be used.

2. Set the coverage metrics to gather

• Right click the file created above, and select 'Copy path'
• Press CTRL + SHIFT + P, and run the 'Cantata: Edit Options File (UI)' command.
• Paste the options file path using CTRL + V into the 'Cantata options file' text box
• Select the coverage types desired by choosing the Rule Set from the drop down menu, or by checking the boxes

3. Change the program build rules to put 'ipg_comp --optfile <path to file created in 1>' in front of the compile command.

• For example, a simple build rule might be changed like this:

"tasks": [
        {
            "type": "cppbuild",
            "label": "C/C++: g++.exe build active file",
-            "command": "C:\\MinGW-8.2\\MinGW\\bin\\g++.exe",
+            "command": "ipg_comp",
            "args": [
+                "--optfile",
+                "${fileDirname}\\ipg.cop",
+                "C:\\MinGW-8.2\\MinGW\\bin\\g++.exe",
                "-g",
                "${file}",
                "-o",
                "${fileDirname}\\${fileBasenameNoExtension}.exe"
            ],
            "problemMatcher": [
                "$gcc"
            ],
            "group": "build",
            "detail": "Builds the currently open file"
        },
       ]

4. Build and run the program, making sure the generated .cov file is placed into the project inside VSCode.

5. Select the source file coverage is wanted on and Run the 'Cantata: Show Coverage' command.

6. Run the 'Cantata: Generate Coverage Report' command to create the report.

Example usage for existing Cantata tests

When Cantata tests are already present for a source file, the coverage obtained by those tests can be viewed in VSCode using this extension.
Simply select the source file inside VSCode and run the 'Cantata: Show Coverage' command. If the Cantata tests for the selected source have been run, and contain coverage, the collected coverage data will then be displayed in the C/C++ editor.
For more information about building and running existing Cantata tests from VSCode, please see the 'Use with VSCode' Cantata Technical Note available from the QA Systems website.

Known Issues

The current release has the following known issues:

1. On Windows, the 'Options File' editor is affected by [Visual Studio Code bug #141080](https://github.com/microsoft/vscode/issues/141080). The right-click context menu clipboard actions do not work for the options file input box. However, the keyboard shortcuts for the actions do work.
2. The options file chosen in the 'Options File' editor is not remembered across sessions, and must be set each time VSCode is started.

Release Notes

1.0.0

Version 1 of the extension providing initial coverage gathering, code annotations and reporting features.