Known Issues | Support | YAML | Policies | Task Parameters | Task Variables | Policy Results | Common Usage Scenarios | FAQ
Build Quality Checks
The Build Quality Checks task allows you to add quality gates to your build process.
Change Notes
You can find the changes notes for this task here.
Known Issues
- We are phasing out Team Foundation Server 2015. If you're still using Team Foundation Server 2015, please stay on version 3.x of the extension which can be downloaded here. For newer versions of Team Foundation Server or Azure DevOps Services please use version 4.x and higher.
Support
If you need help with the extension or run into issues, please contact us at psgerextsupport@microsoft.com or create an issue here.
Adding the Task to a Build Definition
The Build Quality Checks (in task category Build) task needs to be placed after the tasks it should inspect within the same job. If you only need the code coverage policy, you may place the task in a separate job that runs after all other jobs that generate code coverage; code coverage data is aggregated across multiple jobs. In a Visual Studio build definition, e.g., an appropriate place would be after build, test, and symbol indexing/publishing. This ensures that, even if the policy breaks the build, you still get test results as well as the compile output and symbols.
Note: When you want to use the Code Coverage Policy you need to make sure that you publish code coverage values calculated by your test tool first. The Policy itself does not calculate the code coverage. Before you add the task and activate the policy, please make sure that you can already see code coverage values in your build summary. If you are using a test tool other then MSTest (i.e., anything other than the Visual Studio Test task), please use the Publish Code Coverage Results task to publish your coverage data before you add the policy.
Adding the Task to a YAML Build Definition
To add the Build Quality Checks task to a YAML build definition, use the task name and major version like this - task: BuildQualityChecks@9 and set a display name using the displayName property. Then add all task inputs as described under Task Parameters or in the Policies section.
YAML snippet:
- task: BuildQualityChecks@9
displayName: 'Check build quality'
inputs:
# ===== Warnings Policy Inputs =====
#checkWarnings: false # Optional
#warningFailOption: build # Optional; Valid values: build, fixed
#warningThreshold: '0' # Optional
#forceFewerWarnings: false # Optional
#allowWarningVariance: false # Optional
#warningVariance: # Required if allowWarningVariance = true
#showStatistics: false # Optional
#evaluateTaskWarnings: true # Optional
#warningTaskSelectors: '/^((vs|ms)build|ant(\s+.+)?|gradle(w)?(\s+.+)?|grunt|gulp|maven(\s+.+)?|xamarin(android|ios)|xcode(\s+.+)?|cmake|build\s+.+)$/i' # Optional (alias: warningTaskFilters)
#warningSelectors: # Optional (alias: warningFilters)
#inclusiveSelection: false # Optional (alias: inclusiveFiltering)
#warningExclusions: # Optional
#evaluateFileWarnings: false # Optional
#warningFilesFolder: # Optional
#warningFiles: # Required if evaluateFileWarnings = true
#fileWarningSelectors: # Required if evaluateFileWarnings = true (alias: warningFileFilters)
#fileWarningExclusions: # Optional
#warningFilesArtifact: # Required if evaluateFileWarnings = true and (warningFailOption = build or showStatistics = true)
# ===== Code Coverage Policy Inputs =====
#checkCoverage: false # Optional
#coverageFailOption: build # Optional; Valid values: build, fixed
#coverageType: blocks # Optional; Valid values: blocks, lines, branches, custom
#customCoverageType: # Required if coverageType = custom
#treat0of0as100: false # Optional
#coverageThreshold: '60' # Optional
#forceCoverageImprovement: false # Optional
#coverageUpperThreshold: '80' # Optional
#ignoreDecreaseAboveUpperThreshold: true # Optional
#useUncoveredElements: false # Optional
#allowCoverageVariance: false # Optional
#coverageVariance: # Required if allowCoverageVariance = true
#coverageDeltaType: percentage # Optional; Valid values: absolute, percentage
#coveragePrecision: '4' # Optional
#buildConfiguration: # Optional
#buildPlatform: # Optional
#explicitSelector: false # Optional (alias: explicitFilter)
# ===== Baseline Inputs =====
#includePartiallySucceeded: true # Optional
#fallbackOnPRTargetBranch: true # Optional
#baseDefinitionSelector: # Ignored - only used by UI editor
#baseDefinitionId: # Optional
#baseRepoId: # Ignored - only used by UI editor
#baseBranchRef: # Optional
# ===== Reporting Inputs =====
#runTitle: # Optional
#fileAnalysisTitle: # Optional
# ===== Advanced Inputs =====
#disableCertCheck: false # Optional
#createBuildIssues: true # Optional
#addAttachment: true # Optional
Policies
The Build Quality Checks task currently supports two policies (click the link for details):
- Warnings Policy - Allows you to fail builds based on the number of build warnings.
- Code Coverage Policy - Allows you to fail builds based on the code coverage value of your tests.
Task Parameters
Baseline
If you choose Previous Value for the Fail Build On option for one of the policies, the policy value (e.g., number of warnings) is, by default, compared to the corresponding value from the last build that ran for the current build definition. If the build definition targets a Git repository hosted in Azure DevOps Services or Team Foundation Server, the policy will look for the last build that ran against the same branch as the current build. This behavior can be customized with the following parameters:
Include Partially Succeeded Builds: Uncheck this option if policy values should only be compared to successful baseline builds. In most cases, including partially succeeded builds is the best option, so you can use the default setting.
YAML: includePartiallySucceeded - (Optional) Default value is true.
Fall Back on PR Target Branch Disable this option to prevent the task from automatically looking for a baseline build for the pull request target branch if a baseline for the current branch cannot be found. Naturally, the first build that runs for a pull request won't find a baseline build for the current branch, since pull requests run builds on a special branch named refs/pull/<PR-ID>/merge, which is dynamically created by the pull request. To prevent that the first pipeline run for each PR automatically succeeds due to the lack of a baseline, the task automatically looks for a baseline build associated with the pull request's target branch, which usually provides a good quality baseline. If you still want the first PR build to always succeed, disabled this option to prevent the fallback.
YAML: fallbackOnPRTargetBranch - (Optional) Default value is true.
Definition Selector If you have lots of build definitions, it can get hard to find the right one in the Build Definition drop-down list. You can use the Definition Selector to limit the number of definitions shown in the list. Either enter a specific definition name or use the asterisk (*) wildcard to search for definitions starting with (e.g., Def*), ending with (e.g., *def), or containing (e.g., *def*) a specific value. If you want to list all build definitions in your project, use the default value "*".
YAML: baseDefinitionSelector - (Ignored) This parameter is ignored in YAML definitions.
Build Definition: Select the build definition that should be used to search for the baseline build. If you do not set a value, the last build of the current build definition will be used when comparing policy values. If the drop-down list is empty, please click the refresh icon to reload the list of available build definitions. The drop-down list shows a maximum of 1000 build definitions. If your definition is not visible, please enter the build definition ID manually. See TFVC Feature Branches for examples for when using a different build definition might be useful.
YAML: baseDefinitionId - (Optional) Default is empty. Provide a build definition ID.
Note: To always compare policy values to builds from the current build definition that target a specific branch, you need to choose the current build definition here and then select the appropriate values for Repository and Branch (Git).
Repository: Select the repository that is used to search for baseline branches. The drop-down list is populated after you chose the Build Definition and will always contain the repository that the selected build definition is connected to. If the drop-down list is empty after selecting the Build Definition, please click the refresh icon to reload the repository information.
YAML: baseRepoId - (Ignored) This parameter is ignored in YAML definitions.
Note: When you change the Build Definition after selecting a repository, you might see a GUID value in the repository parameter. This is a refresh limitation of the build UI. Please select the repository again to correct this.
Branch (Git): Select the branch that should be used to search for the baseline build. If you do note set a value, the last build targeting the currently built branch will be used when comparing policy values, unless the build runs in the context of a pull request. In that case, the task looks for the last build targeting the pull request target branch if it cannot find a build for the currently built branch. See Pull Request Policy for more information.
If the drop-down list is empty after selecting the Repository, please click the refresh icon to reload the list of available branches. Branches are shown with their Git ref name, e.g. refs/heads/master or refs/heads/myTopicBranch.
YAML: baseBranchRef - (Optional) Default is empty. Provide a Git branch ref (e.g., refs/heads/master).
Note: When you change the Build Definition and Repository after selecting a branch, you need to select the branch again. Otherwise, a wrong branch name may be saved to the task configuration. This is a refresh limitation of the build UI.
Note: Whenever you choose a different baseline branch, make sure that the retention policy is configured to keep at least one successful build for your baseline branch!
Reporting Options
Run Title: If you are using multiple Build Quality Checks tasks within the same build, you may use this parameter to specify a title that is associated with a specific instance of the task. This will help distinguishing between the task results in the summary section. The run title is used as the section header in the summary (see Extension Summary Section).
YAML: runTitle - (Optional) Default is empty.
Note: Run titles cannot contain the following characters: ", <, >, |, :, ;, *, ?, \, /, %, +, #, [, ]. If the configured value does contain invalid charachters, they are replaced by an underscore.
File Analysis Title: Provide a custom name for the file warnings analysis. The name is used in warning statistics instead of the default value File Analysis. (see Show Warning Statistics).
YAML: fileAnalysisTitle - (Optional) Default is empty.
Advanced
Disable NodeJS certificate check: Check this option if your Team Foundation Server is using a self-signed or corporate SSL certificate and your build agent version is lower than 2.117.0. The option disables the certificate chain validation of NodeJS. Please read here for details.
YAML: disableCertCheck - (Optional) Default is false.
Log task results as build issues: When enabled, warnings and errors emitted by the task are also logged as regular build warnings/errors. This makes them visible on the build summary page. If this option is disabled, warnings and errors are only displayed in the task summary section. Note: The option does not impact the task result. Even if it is disabled, the task will still fail when one of the policies is violated. This option is enabled by default.
YAML: createBuildIssues - (Optional) Default is true.
Add policy results attachment: When enabled, policy results are published as build attachment, so they can be displayed in the build summary. If this option is disabled, the Extensions tab in the build summary UI is not displayed. In addition, warning statistics will be disabled, since they are only visible through the build summary UI. In most cases, you should not disable this option. Note: The option does not impact the task result. Even if it is disabled, the task will still fail when one of the policies is violated. This option is enabled by default.
YAML: addAttachment - (Optional) Default is true.
Task Variables
In addition to parameters visible in the task UI there are a few variables you can set to affect the tasks behavior:
BQC.ForceNewBaseline: You can set this variable (usually when queueing a new build) to true to force the policies to ignore an existing baseline build. This essentially creates a new baseline for future builds. This is especially helpful if there was a larger refactoring or redesign that resulted in a big change in code coverage and you need to establish a new base value. The default value is false.
BQC.LogRawData: You can set this variable to true to enable logging of raw JSON data read by the task. In most cases you should not use this option as it does increase the log size of the task and may show sensitive data in the log file. However, this data is very helpful in support scenarios. Note that this variable only takes effect if the variable System.Debug is also set to true. The default value is false.
BQC.Statistics.MaxAddedWarnings: Set this variable to the maximum number of added warnings that should be listed in the warnings statistics. The default value is 50.
BQC.Statistics.MaxRemovedWarnings: Set this variable to the maximum number of removed warnings that should be listed in the warnings statistics. The default value is 50.
BQC.DisableVariableCreation: Set this variable to true to disable the creation of output variables. This can be done to work around an issue that currently exists on Team Foundation Server 2017. The default value is false.
Policy Results
Policy results are returned in various ways to make it easy for you to see and use them in different scenarios:
Build Issue List
If the advanced option Log task results as build issues is enabled (default), the task reports policy violations as regular build issues. These are listed with all other issues on the build's summary page:
Extension Summary Section
The Build Quality Checks task creates its own summary section in the build summary view. This section displays all success, warning, and error messages for all activated policies. If you run a multi-configuration build, a subsection for each build job is created so you can see exactly which configuration might have quality issues.
Note: Please see the Limitations and Special Cases section of the Code Coverage Policy for possible issues with multi-configuration builds and code coverage.
Policy Result Variables
In addition, each policy you run creates an output variable that you can use to conditionally run other tasks in your pipeline (e.g., create a technical debt work item if quality metrics get worse). See policy documentation for more information.
Pull Request Statuses
If you run the task as part of a pull request validation build, each policy also publishes its result as a pull request status:
See Pull Requests and TFVC Feature Branches for more information.
Important: If you want to use the pull request features of the task, please ensure that your build account has the Contribute to pull requests permission! See Git repository permissions for more information. Depending on your build job authorization scope you need to either give that permission to your project build account (e.g., YourProject Build Service (YourOrganization)) or your project collection build account (e.g., Project Collection Build Service (YourOrganization)).
Common Usage Scenarios
FAQ
We have put together a list of frequently asked questions and answers in our FAQ document. If you feel we need to add a specific question to the list, feel free to send it to our support address.
Checklist board icon | Icon designed by Vexels.com
