Datadog Continuous Testing for Azure DevOps

Overview

With the SyntheticsRunTests task, you can run Synthetic tests within your Azure Pipeline configuration and ensure all your teams using Azure DevOps can benefit from Synthetic tests at every stage of the software lifecycle.

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

Authentication

Service Connection

To connect to your Datadog site, Datadog recommends setting up a custom service connection when configuring the SyntheticsRunTests task.

You need to provide the following inputs:

Datadog site: Your Datadog site. The possible values are listed in this table.
Custom subdomain (default: app): The custom subdomain to access your Datadog organization. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.
API key: Your Datadog API key. This key is created in your Datadog organization.
Application key: Your Datadog application key. This key is created in your Datadog organization.

API and Application keys

API key: Your Datadog API key. This key is created in your Datadog organization and should be stored as a secret.
Application key: Your Datadog application key. This key is created in your Datadog organization and should be stored as a secret.
Datadog site: Your Datadog site. The possible values are listed in this table.
Custom subdomain (optional): The custom subdomain to access your Datadog organization. If your URL is myorg.datadoghq.com, the custom subdomain is myorg.

Setup

To connect to your Datadog account, create a Datadog CI service connection in your Azure pipelines project. Once created, all you need is the name of the service connection in the tasks.

Install the Datadog Continuous Testing extension from the Visual Studio Marketplace in your Azure Organization.
Add your Datadog API and application keys in the Datadog CI service connection, or as secrets to your Azure Pipelines project.
In your Azure DevOps pipeline, use the SyntheticsRunTests task.

Your task can be simple or complex.

Simple usage

Example task using public IDs

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    publicIds: |
      abc-d3f-ghi
      jkl-mn0-pqr

Example task using existing `synthetics.json` files

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    files: 'e2e-tests/*.synthetics.json'

For an example test file, see this test.synthetics.json file.

Example task using pipeline secrets for authentication

- task: SyntheticsRunTests@1
  inputs:
    authenticationType: 'apiAppKeys'
    apiKey: '$(DatadogApiKey)'
    appKey: '$(DatadogAppKey)'
    datadogSite: '$(DatadogSite)'
    subdomain: 'myorg'

Complex usage

Example task using the `testSearchQuery`

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'

Example task using the `testSearchQuery` and variable overrides

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    connectedService: 'my-datadog-ci-connected-service'
    testSearchQuery: 'tag:e2e-tests'
    variables: |
      START_URL=https://staging.website.com
      PASSWORD=$(StagingPassword)

Example task using a global configuration file with `configPath`

By default, the path to the global configuration file is datadog-ci.json. You can override this path with the config_path input.

- task: SyntheticsRunTests@1
  displayName: Run Datadog Synthetic tests
  inputs:
    authenticationType: 'connectedService'
    configPath: './global.config.json'
    connectedService: 'my-datadog-ci-connected-service'

Inputs

For more information on the available configuration, see the datadog-ci synthetics run-tests documentation.

Name	Description
`apiKey`	Your Datadog API key. This key is created in your Datadog organization and should be stored as a secret. _{Required when authenticationType == apiAppKeys}
`appKey`	Your Datadog application key. This key is created in your Datadog organization and should be stored as a secret. _{Required when authenticationType == apiAppKeys}
`authenticationType`	(Required) How to store and retrieve credentials. _{Must be either apiAppKeys or connectedService}
`batchTimeout`	Specifies the timeout duration in milliseconds for the CI batch. When a batch times out, the CI job fails and no new test runs are triggered, but ongoing test runs complete normally. _{Default: 1800000 (30 minutes)}
`connectedService`	The name of the Datadog CI service connection. _{Required when authenticationType == connectedService}
`configPath`	The path to the global configuration file that configures datadog-ci. _{Default: datadog-ci.json}
`datadogSite`	Your Datadog site. The possible values are listed in this table. _{Default: datadoghq.com}
`failOnCriticalErrors`	Fail the CI job if a critical error that is typically transient occurs, such as rate limits, authentication failures, or Datadog infrastructure issues. _{Default: false}
`failOnMissingTests`	Fail the CI job if the list of tests to run is empty or if some explicitly listed tests are missing. _{Default: false}
`failOnTimeout`	Fail the CI job if the CI batch fails as timed out. _{Default: true}
`files`	Glob patterns to detect Synthetic test configuration files, separated by new lines. _{Default: {,!(node_modules)/*/}.synthetics.json}
`jUnitReport`	The filename for a JUnit report if you want to generate one. _{Default: none}
`locations`	Override the list of locations to run the test from, separated by new lines or commas. The possible values are listed in this API response. _{Default: none}
`publicIds`	Public IDs of Synthetic tests to run, separated by new lines or commas. If no value is provided, tests are discovered in Synthetic test configuration files. _{Default: none}
`selectiveRerun`	Whether to only rerun failed tests. If a test has already passed for a given commit, it is not rerun in subsequent CI batches. By default, your organization's default setting is used. Set it to `false` to force full runs when your configuration enables it by default. _{Default: none}
`subdomain`	The custom subdomain to access your Datadog organization when `authenticationType == apiAppKeys`. If your URL is `myorg.datadoghq.com`, the custom subdomain is `myorg`. _{Default: app}
`testSearchQuery`	Use a search query to select which Synthetic tests to run. Use the Synthetic Tests list page's search bar to craft your query, then copy and paste it. _{Default: none}
`variables`	Override existing or inject new local and global variables in Synthetic tests as key-value pairs, separated by new lines or commas. For example: `START_URL=https://example.org,MY_VARIABLE=My title`. _{Default: none}

Outputs

Name	Description
`batchUrl`	The URL of the batch.
`criticalErrorsCount`	The number of critical errors.
`failedCount`	The number of failed results.
`failedNonBlockingCount`	The number of failed non-blocking results.
`passedCount`	The number of passed results.
`previouslyPassedCount`	The number of previously passed results.
`testsNotFoundCount`	The number of not found tests.
`testsSkippedCount`	The number of skipped tests.
`timedOutCount`	The number of timed out results.
`rawResults`	The list of results, as a raw JSON string.

Datadog Continuous Testing

Datadog