DataFactorytools(in Preview)

This Extensions provides functionality to generate and validate an ARM template, start and stop triggers, and clean unused resources after deploy.

wiki: https://wikicorp.telefonica.com.br/display/DEVELOP/DataFactorytools

Note: Please use node version 18.x to avoid failures while running the validate command, as it uses Microsoft Library.

DataFactorytools task!

This extension contributes with 2 Tasks!

• BuildAzureDataFactory
• PrePostDeployDataFactory

Recursos

• Validation of the json files of the Azure Devops resources. Verification of the integrity and correction of the JSON files that define the resources in Azure DevOps. This ensures that the files are well formatted and error-free.

• Export of Azure DevOps resources to an ARM (Azure Resource Manager) model with the possibility of selecting directories for each resource. This is especially useful for resources related to the Azure Data Factory (ADF) infrastructure, such as Integration Runtimes and Linked Services.

• Selective export of Azure Data Factory (ADF) resources in an ARM model, allowing you to define a specific folder in the ADF where Author resources (such as pipelines, datasets, and dataflows) have been added. Only the resources of this folder and its dependencies will be exported. It is important to note that the folder must have the same name in all resources.

• Stop and Start the triggers that are running from the ADF. To avoid errors during the deployment of the template in Azure Data Factory (ADF), it is recommended to stop the triggers that are running.

• Perform cleanup of the resources after the deployment of the ADF ARM Template, If the export was selective, the folder should be informed via the authorResourceDeleteFolder parameter so that the task performs the cleaning only of the resources of this folder.

Observações

• The validation uses the Microsoft azure-data-factory-utilities module, which is only compatible with Node 18. This module can be installed in a previous task or, if it is not present in the working directory or in the folder specified in this task, this task will install it:

Exemplo de uso

- job:
  ...
  steps:
        - task: BuildAzureDataFactory@0
          inputs:
            action: 'build'
            subscriptionId: '000000-000000-000000-00000'
            resourceGroup: 'rg-test'
            factoryName: 'adf-test'

        - task: BuildAzureDataFactory@0
          inputs:
            action: 'validate'
            subscriptionId: 'cd4ed2c4-eb6d-48d7-98b7-e6db50218b7'
            resourceGroup: 'rg-resourceinv-prodlike-test'
            factoryName: 'adf-resourceinv-prodlike-test'

        - task: PrePostDeployDataFactory@0
          inputs:
            action: 'postdeployment'
            deleteResources: true
            serviceConnectionId: 'ACR-TEST'
            subscriptionId: '000000-0000-0000-0000-000000000'
            resourceGroup: 'rg-test'
            dataFactoryName: 'adf-test'
            armTemplatePath: '$(Build.SourcesDirectory)/ARMTemplateForFactory.json'

        - task: PrePostDeployDataFactory@0
          displayName: "Stop Triggers"
          inputs:
            action: 'predeployment'
            serviceConnectionId: 'ACR-ALETEIA-TEST'
            subscriptionId: '000000-0000-0000-0000-000000000'
            resourceGroup: 'rg-test'
            dataFactoryName: 'adf-test'
            armTemplatePath: '$(Build.SourcesDirectory)/ARMTemplateForFactory.json'
        
        - task: PrePostDeployDataFactory@0
          inputs:
            action: 'postdeployment'
            deleteResources: true
            serviceConnectionId: 'ACR-ALETEIA-TEST'
            subscriptionId: 'cd4ed2c4-eb6d-48d7-98b7-e6db50218b71'
            resourceGroup: 'rg-resourceinv-prodlike-test'
            dataFactoryName: 'adf-resourceinv-prodlike-test'
            armTemplatePath: '$(Build.SourcesDirectory)/ARMTemplateForFactory.json'
            authorResourceDeleteFolder: 'test_folder'