General Description

Azure Pipelines task facilitates an integration between Azure DevOps Pipeline (Build / Release) and Azure Data Explorer (ADX) databases. This extension includes 3 basic tasks:

• Azure Data Explorer Command - Run Admin Commands against ADX cluster
• Azure Data Explorer Query - Run Queries against ADX cluster and parse the results
• Azure Data Explorer Query Server Gate - Agentless task to Gate releases depending on the query outcome

The content and functionality of the tasks can be taken from either a git source control or build sources or build artifacts or inline the task. The tasks authenticates with AAD App details (ID / Key) either from variabes or an Azure DevOps Service endpoint.

Official Azure Data Explorer - Azure DevOps integration

Details

Azure Data Explorer Endpoint

Enter your Cluster URL, AppID, AppKey and TenantID in the Service connection appropriate Fields (Authentication Token Field should be left empty)

To create a new service connection go to project settings page (the gear icon in the lower lefthand side)

*** if you have service endpoints created before version 1.7.1 you might need to recreate them for the server gate task to work properly ***

Admin command task

Add your endpoint information (Cluster and Database) to Endpoint URLs (optional: use values from build variables)

Add your AppID, AppKey and TenantID (Use Variable Group or Secret Build Variable) the Resource URI is the service endpoint providing the jwt token requested for accesing your cluster should be the base URL for your cluster,

Or you can Use an Azure Data Explorer Service Endpoint
Check the "Use Service Endpoint" Checkbox to select an existing Azure Data Explorer service connection

if you don't already have a Service connection configured, click the "Manage" link to create a new service connection

Add the match pattern for your *.csl files from the Source Control (for running multiple commands in the same task)

***the single line option allows having multiple files, with one command per file ***

Alternatively: switch to an inline script and write your command directly in the task (no empty lines - one command per task)

OR: Add you file directly from a git repository path

In case the command is a long running asynchronous operations you can check "Wait for long Async Admin commands to complete"

checkbox to have the task run ".show operations " in the REST response, and the task will wait for the command to complete pass or fail based on the result (Task will fail if any command in the script is not async - Use only on async commands)

Optional: Add the name of the output variable (Or Path to output file) you want to command response to be stored in (and use it in downstream tasks) if you run multiple commands only last query is saved, when "Save only last response is unchecked multiple variables will be created (one foreach command and endpoint - with ' prefix' for filenames or ' suffix' for variables)"

Yaml Sample Usage

steps:
- task: Azure-Kusto.ADXAdminCommands.PublishToADX.ADXAdminCommand@1
  displayName: '<Task Name>'
  inputs:
    script: '<inline Script>'
    waitForOperation: true
    kustoUrls: '$(CONNECTIONSTRING):443?DatabaseName=""'
    customAuth: true
    connectedServiceName: '<Service Endpoint Name>'
    serialDelay: 1000
  continueOnError: true
  condition: ne(variables['ProductVersion'], '') ## Custom condition Sample

Query task

Add your endpoint information (Cluster and Database) to Endpoint URLs (optional: use values from build variables)

Add your AppID, AppKey and TenantID (Use Variable Group or Secret Build Variable) the Resource URI is the service endpoint providing the jwt token requested for accesing your cluster should be the base URL for your cluster,

Or you can Use an Azure Data Explorer Service Endpoint
Check the "Use Service Endpoint" Checkbox to select an existing Azure Data Explorer service connection

if you don't already have a Service connection configured, click the "Manage" link to create a new service connection

Add the match pattern for your *.csl files from the Source Control (for running multiple commands in the same task)

***the single line option allows having multiple files, with one command per file ***

Alternatively: switch to an inline script and write your command directly in the task (no empty lines - one command per task)

OR: Add you file directly from a git repository path

Query Exit Criteria: you can choose to fail the task based on the response record (rows) count

or based on the response (single) value (make sure the query only returns a single record (row and field))

Optional: Add the name of the output variable (Or Path to output file) you want to command response to be stored in (and use it in downstream tasks) if you run multiple commands only last query is saved, when "Save only last response is unchecked multiple variables will be created (one foreach command and endpoint - with ' prefix' for filenames or ' suffix' for variables)"

Server Gate

Release gates documentation

Once we go to Release Definition, by accessing pre/post approval setting we can enable Gates and add Azure Data Explorer Query as a Gate

We can query ADX using Inline query

We can query ADX using File path

How to add an endpoint to be used by Azure Data Explorer Query Gate or Task

Getting ApplicationId and ApplicationKey

How to add Query Kusto as a Task

Input parameters

• Service endpoint: Select an ADX endpoint that should be used to connect to Kusto, to execute the query. Check section How to add an endpoint to be used by ADX Query Gate or Task to add an ADX endpoint.

• Database name: ADX database name to run the query. example: vso or vsodev.

• Type: Query can be taken from a repository file path or inline.

  Parameters for Inline query

  • Inline query: You can write your Kusto query. More info on Kusto query language.

  Parameters for File path

  • Repository name: Repository name in which query file exists.

  • Branch name: Branch name in which query file exists. example, master.

  • Query file full path: Query (csl) file full path in the given branch. example, /MyKustoQueries/KustoQuery.csl.

• Maximum threshold: The maximum number of rows from the query result.

• Minimum threshold: The minimum number of rows from the query result.

How to adjust threshold

Max and min threshold are output rows expected from the Kusto query

Yaml Sample Usage

steps:
- task: Azure-Kusto.ADXAdminCommands.ADXQuery.ADXQuery@1
  displayName: '<Task Display Name>'
  inputs:
    script: |  
     let badVer=
     RunnersLogs | where Timestamp > ago(30m)
         | where EventText startswith "$$runnerresult" and Source has "ShowDiagnostics"
         | extend State = extract(@"Status='(.*)', Duration.*",1, EventText)
         | where State == "Unhealthy"
         | extend Reason = extract(@'"NotHealthyReason":"(.*)","IsAttentionRequired.*',1, EventText)
         | extend Cluster = extract(@'Kusto.(Engine|DM|CM|ArmResourceProvider).(.*).ShowDiagnostics',2, Source)
         | where Reason != "Merge success rate past 60min is < 90%"
         | where Reason != "Ingestion success rate past 5min is < 90%"
         | where Reason != "Ingestion success rate past 5min is < 90%, Merge success rate past 60min is < 90%"
         | where isnotempty(Cluster)
         | summarize max(Timestamp) by Cluster,Reason 
         | order by  max_Timestamp desc      
         | where Reason startswith "Differe"
         | summarize by Cluster
     ;   
      DimClusters | where Cluster in (badVer)
     | summarize by Cluster , CmConnectionString , ServiceConnectionString ,DeploymentRing
     | extend ServiceConnectionString = strcat("#connect ", ServiceConnectionString)
     | where DeploymentRing == "$(DeploymentRing)"
    kustoUrls: 'https://<ClusterName>.kusto.windows.net:443?DatabaseName=<DataBaneName>'
    customAuth: true
    connectedServiceName: '<Service Endpoint Name>'
  continueOnError: true

Run the task in a CI pipeline and see the JSON results in the log,
or, alternatively, get it in downstream tasks with the Output Variable $(OutputVariable)
You can save it to file or parse it with JSON parsing tool

Contributions

This extension is maintained by Kusto Ops Team Publisher Page

Microsoft docs

Official Microsoft Documentation Site for Azure Data Explorer

Github

Developer Private Fork
Official Azure Pipeline tasks