Test Case Association

What does this extension do?

This extension is responsible for populating the "Automation" fields attached to Azure Devops test cases work items. By populating these fields you are able to create the association between Azure Devops your automated tests in C#. Maintaining this association is important for two reasons:

• Document what your automated tests cases are supposed to be validating
• Kick off automated test runs via Azure Devops Test Plan

How do I use it?

WARNING: This extension currently only supports Xunit.
WARNING: This extension was developed to be used in two seperate build pipelines per project: one for PRs, and the other for merges to master. You may still use this extension if you are using a single build pipeline for your project, but the integration may have additional friction.

Setting your test case IDs:

Install the nuget package "TestCaseAttributes" into your project and attach the TestCaseId method attribute to your test methods. Use the work item ID from Azure Devops inside the TestCaseId attribute on your test methods.

using TestCaseAttributes.TestCaseId;

[Fact]
[TestCaseId("1234")]
public void YourTest():
    // your test logic/assertion

Configure your PR build Pipeline

There should be a build pipeline that is triggered when pull requests are opened. For this pipeline enable the Dry Run flag in the TestCaseAssociation task configuration.
This task is going to validate multiple things and fail the build if the following validations fail:

• Test case IDs attached to the TestCaseId attributes are valid Test Case work items in Azure Devops
• No TestCaseId values are duplicated
• Number of automated tests without the TestCaseId attribute is exceeded (this threshold is configurable, see "Max Missing Test Case Ids" in the task configuration)
  Set a secured environment variable to your build pipeline:

AZURE_TOKEN=<your_personal_access_token>

Example yaml configuration:

steps:
- task: GoatWranglers.build-release-task.custom-build-release-task.runtestcaseassociation@0
  displayName: 'Test Case Associations'
  inputs:
    AZURE_HOST: 'https://goatwranglers.visualstudio.com'
    AZURE_PROJECT: GWranglers
    TEST_DLL: GoatWranglers.dll
    MAX_MISSING_TEST_CASE_IDS: 50

Configure your master build Pipeline

This build pipeline should be triggered any time a pull request is merged into master. For this pipeline disable the Dry Run flag in the TestCaseAssociation task configuration.
This task will reach out to the Azure Devops APIs to create the associations by populating the automation fields on the Test Case work item.
Set a secured environment variable to your build pipeline:

AZURE_TOKEN=<your_personal_access_token>

Example yaml configuration:

steps:
- task: GoatWranglers.build-release-task.custom-build-release-task.runtestcaseassociation@0
  displayName: 'Test Case Associations'
  inputs:
    DRY_RUN: false
    AZURE_HOST: 'https://goatwranglers.visualstudio.com'
    AZURE_PROJECT: GWranglers
    TEST_DLL: GoatWranglers.dll
    MAX_MISSING_TEST_CASE_IDS: 50

Create your release pipeline to execute tests via test plan

1. Create a new Release Pipeline in Azure Devops
2. Add the artifact
   1. The source for this artifact should consume from your master build pipeline for the project that contains the tests you wish to execute.
   2. The artifact does not need to be on a schedule
3. Add a stage named "Run tests"
   1. For Pre-Deployment Conditions, select "Manual Only"
4. Add the following tasks to the "Run Tests" stage
   1. Add a task to restore your project (dotnet restore)
   2. Add the following task "Visual Studio Test Platform Installer"
   3. Add the following task "Visual Studio Test"
      1. Task Version 2.*
      2. Select Tests Using: Test Run
      3. Select Test platform version: Installed by Tools Installer
         See this how-to article for more information.

Create your Azure Devops test plan

1. Create a new Test Plan in Azure Devops
2. Right click your Test Plan and select "Test Plan Settings"
   1. Set the Build Pipeline as your master build pipeline and consume from latest
   2. Set the Release Pipeline as the release pipeline to the pipeline created in the previous step and consume the "Run Tests" stage
3. Add a Query-based Suite with the following query:

   Work Item Type, In Group, Microsoft.TestCaseCategory
   Automated Test Storage, =, <your_test_dll> 
   

4. Right click the Test Suite and select "Run for web application"
5. At this point, the Release Pipeline should have a new build and the results may be viewed in "Test Plans -> Runs" See this how-to article for more information.