PublishTestAttachments

	Status
Unit tests
Integration tests

Introduction

PublishTestAttachments is an Azure DevOps extension useful to attach extra files and informations to tests being run as part of an Azure Pipeline.

The extension exposes a new task which can be included in your YAML or classic pipeline and will attach content produced by the test execution in the corresponding attachments tab of the test report. For example:

How to use it

The PublishTestAttachments task has to be run after the test run report was published, otherwise it won't find any test runs and tests results to look for. A typical use case is:

1. Run the tests using the testing framework of your choice. During the execution of the test, be sure to dump down the files you want to reports in case the test fails.
2. Publish the test run report using the built-in PublishTestResults task. This should run with the succededOrFailed() condition so that the test run is published even if tests failed.
3. Add the PublishTestAttachments task in the same job, since it needs to access the files written by the tests. This should run with the succededOrFailed() condition so that the attachments are published even if tests failed.

Configure the task

The task design is all around finding the attachment files of a given test in a folder on an expected location. If the task won't be able to find the expected folder, it will skip that test. In the task parameters, we use the notion of capturing variables in order to properly derive the expected folder with the attachments. We use the following syntax to represent such variables: @(myVariable)

The task accepts the following parameters:

• pathPattern: Required. The pattern for deriving the path of the folder to check for each test.
• testPattern: Optional. The pattern for decomposing the test name into different capturing variables. It defaults to '@(testName)' which means the entire test name is assigned to the testName variable
• testRunFilter: Optional. Useful for parallel test jobs. If set, only tests belonging to the passed run title will be taken into consideration. It assumes the test runs titles are unique in each build.

Capturing variables used in pathPattern must be defined in testPattern. The testClass variable is already available for usage in pathPattern and pointing to the location of the test function/method (matches the "Test file" in DevOps test report)

Examples

Simple usage

- task: PublishTestResults@2
  condition: succeededOrFailed()
  inputs:
    testResultsFormat: JUnit
    testResultsFiles: '**/test-reports/TEST-*.xml'
    testRunTitle: My test run
- task: PublishTestAttachments@0
  condition: succeededOrFailed()
  inputs:
    pathPattern: test-errors/@(testClass)/@(testName)
    testRunFilter: My test run

This is the simplest usage of the task. There's no decomposition of the test name and path is simply derived from test class and test name, plus some constant. Given the following tests:

com.example.foo.TestCaseOne.test_one
com.example.foo.TestCaseOne.test_two
com.example.foo.bar.TestCaseThree.test_alpha

For each of those, the task will look for the existence of the following folders, respectively:

test-errors/com/example/foo/TestCaseOne/test_one
test-errors/com/example/foo/TestCaseOne/test_two
test-errors/com/example/foo/bar/TestCaseThree/test_alpha

If a folder is found, then its content will be attached to the corresponding test

Test name decomposition

- task: PublishTestResults@2
  condition: succeededOrFailed()
  inputs:
    testResultsFormat: JUnit
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    testRunTitle: My test run
- task: PublishTestAttachments@0
  condition: succeededOrFailed()
  inputs:
    testPattern: '@(testName) @(browser)-@(resolution)'
    pathPattern: target/test-errors/@(testClass)/@(testName)/@(browser)/@(resolution)
    testRunFilter: My test run

This configuration will enable test name decomposition over 3 different variables. It also assumes each test name will follow the specified pattern. If a given test name does not follow the pattern and a decomposition is not possible, that test will be ignored from the attaching process.

Given the following tests:

com.example.foo.TestCaseOne.test_one Firefox-VGA
com.example.foo.TestCaseOne.test_two Firefox-HD
com.example.foo.bar.TestCaseThree.test_alpha OtherBrowser-FULL_HD

For each of those, the task will look for the existence of the following folders, respectively:

target/test-errors/com/example/foo/TestCaseOne/test_one/Firefox/VGA
target/test-errors/com/example/foo/TestCaseOne/test_two/Firefox/HD
target/test-errors/com/example/foo/bar/TestCaseThree/test_alpha/OtherBrowser/FULL_HD

Supported test frameworks

Used testing framework must be able to produce a report supported by Azure DevOps (e.g. junit, xunit, nunit). Although the extension itself does not set constraints against specific test tooling (except the above one), the current testing coverage is limited to:

• pytest
• Maven

However, this list can grow in the future if there's interest on some particular testing framework.

License

PublishTestAttachments is licensed under the MIT license.