Azure DevOps AI Evaluation Report
This extension is designed to work with the reports created by the Microsoft.Extensions.AI.Evaluation.Reporting
library. This extension contains a task for publishing the data along with an extension that will display the report in a tab in your build pipeline results.
Publishing the data
The evaluation data must be in the JSON format generated by the AI Evaluation libraries. You can use either the Microsoft.Extensions.AI.Evaluation.Reporting
library directly or the Microsoft.Extensions.AI.Evaluation.Console
tool to generate this data.
This extension publishes a PublishAIEvaluationReport
task that will publish the data for you. Having this task in your pipeline also triggers the report tab to appear.
- task: PublishAIEvaluationReport@0
displayName: 'Publish AI Evaluation Data'
inputs:
reportFile: '$(build-artifacts)\report.json'
If your pipeline restricts uploading attachments from a task, you can disable the attachment upload feature and use a different method to upload the report data. You should still include PublishAIEvaluationReport
task as it will trigger the display of the reporting tab. When using this method, the attachment must be uploaded to the pipeline with type=ai-eval-report-json
and name=ai-eval-report
.
For example,
- task: PublishAIEvaluationReport@0
displayName: 'Display AI Evaluation Data'
inputs:
reportFile: '$(build-artifacts)\report.json'
disableAttachmentUpload: true
- script: |
echo "##vso[task.addattachment type=ai-eval-report-json;name=ai-eval-report;]$(build-artifacts)\report.json"
The Microsoft.Extensions.AI.Evaluation libraries
Microsoft.Extensions.AI.Evaluation
is a set of .NET libraries defined in the following NuGet packages that have been designed to work together to support building processes for evaluating the quality of AI software.
Install the packages
From the command-line:
dotnet add package Microsoft.Extensions.AI.Evaluation
dotnet add package Microsoft.Extensions.AI.Evaluation.Quality
dotnet add package Microsoft.Extensions.AI.Evaluation.Reporting
Or directly in the C# project file:
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.AI.Evaluation" Version="[CURRENTVERSION]" />
<PackageReference Include="Microsoft.Extensions.AI.Evaluation.Quality" Version="[CURRENTVERSION]" />
<PackageReference Include="Microsoft.Extensions.AI.Evaluation.Reporting" Version="[CURRENTVERSION]" />
</ItemGroup>
You can optionally add the Microsoft.Extensions.AI.Evaluation.Reporting.Azure
package in either of these places if you need Azure Storage support.
dotnet tool install Microsoft.Extensions.AI.Evaluation.Console --create-manifest-if-needed
Usage Examples
For a comprehensive tour of all the functionality, concepts and APIs available in the Microsoft.Extensions.AI.Evaluation
libraries, check out the API Usage Examples available in the dotnet/ai-samples repo. These examples are structured as a collection of unit tests. Each unit test showcases a specific concept or API, and builds on the concepts and APIs showcased in previous unit tests.
Feedback & Contributing
We welcome feedback and contributions in our GitHub repo.