Azure DevOps Testing Hub Extension to support Non Dotnet Technologies

Overview

This Build/Release Task is mainly for supporting creating Azure DevOps tests from your BDD/Feature Files irrespective of technology (dotnet or non-dotnet), Automated Test Automation association (for dotnet projects) and lastly support for updating test results to specified Test Plan/Test Suites for dotnet and non-dotnet technologies e.g. JavaScript/Python/Java etc. It provided similar test result management as DevOps provides for dotnet automated tests.

What is new in this version?

This version includes some major changes as given below:

• Support for dotnet 8.0
• Fixes for Tag creation even if Azure DevOps tests are not having tags
• Now task can run on any agent irrespective of platform e.g. oSX, Linux and Windows
• Most important, now you can publish the test results to Azure Devops Test Plans for dotnet tests as well, this will help in case we are running tests on Linux agent where VStest task doesn't work.

Usage

• Install the extension in your Azure DevOps organization
• Create/update a pipeline (build or release)
• Go for adding pipeline tasks, and search for "Azure DevOps Testing Extension" and click Add
• Follow steps given below for configuring the task for ccreating tests or updating test results

Configuration Options

This task has three main configurations: Select the Operation: To start with we need to select an Operation type either "Create Tests from Feature Files" or "Update Test Results in Test Plan/Suites"

• Create Tests From Feature Files: This option is used to create test cases in Azure DevOps, following are the mandatory data to provide to support this operation:

  • Folder Containing the Feature Files - For dotnet project make sure to provide path for folder containing your project, for others e.g. Java/JS etc provide the parent folder path contaning feature files.
  • Additional Build Arguments Dotnet Tests - This option is used for providing additional build argument for building the dotnet project for getting the associated automation names. Default value for this option is '--configuration Debug' but can have multiple other dotnet arguments as well.

• Update Test Results in Test Plan/Suites: Use this option to update your test execution results to provided test plan/suite based on given test configuration for non dotnet project(s) only. We need to provide following details with this option:

  • Folder containing the XML Result File - This is the folder containing the test result XML files in JUNit format.
  • Naming Convention For XML Result File - Naming conventions to filter the XML file from above folder e.g. if your files are generated with TestResults-xxxxxx.xml, then use "TestResults" here as filter value.
  • Test Plan - Provide ID of Azure Test Plan to publish test results from above XML files
  • Test suites (comma separated) - Provide a comma separated Ids of test suites under above test plan for search and update test results
  • Test configuration - Test Configuratio ID for updating the tests

• General Settings: This setting is for capturing following details for Create Tests or Update Test Results:

  • DevOps URL - URL fo your organization
  • Your Project Name in DevOps - Name of your project where you want the test cases to be create or containing the Test PLan/Suite to update test results
  • Personal Access Token - It should a PAT with Work Item Management Read/Write access and Test Management Read/Write Access. Make sure to use it as secret variable or key vault
  • Path for dotnet Exe on the agent running the Job - Default value is set, update if your dotnet is at different location
  • Name of your repo containing your test code - Default is empty. Note: Adding any value here will be added as prefixed to the created tests.
  • Is it a Dotnet Project? - This option needs to Set to "True" if your project is a dotnet else set it to "false" for other projects.

Important Note

For publishing the test results in Azure DevOps test plan/suite, with latest release of this extension it supports Windows and Linux agents and also publish test results for Dotnet tests as well. Following are main mandatory steps for projects:

• For non-dotnet projects, projects should be generating the test result in JUnit result XML format.
• For non-dotnet projects, make sure "testcase" element in result XML must match the name of test case title in Azure DevOps to identify the tests.
• For Dotnet Projects, publishing the test results to Azure DevOps Test plans (useful for Linux agents), Add "JunitXml.TestLogger" nuget package to your project for generating test results in JUnit XML format. Also in your pipeline or local run the test like "dotnet --logger:"junit;LogFilePath=test-result.xml"