Use an LLM to review Pull Requests for Azure DevOps

A task for Azure DevOps build pipelines to add GPT / OpenAI-compatible endpoint as PR reviewer

This is a fork of https://github.com/mlarhrouch/azure-pipeline-gpt-pr-review. We're doing some changes to make the extension more flexible, but there's more work to be done. The primary reason for the fork is that we wish to use it with our self-hosted ollama server, which the original extension does not support.

Setup

Before using this task, ensure that the build service has permissions to contribute to Pull Requests in your repository, and allow the task to access the system token.

Give permission to the build service agent

Allow Task to access the system token

YAML pipelines

Add a checkout section with persistCredentials set to true.

steps:
  - checkout: self
    persistCredentials: true

Classic editors

Enable the option "Allow scripts to access the OAuth token" in the "Agent job" properties :

Azure Open AI service

If you choose to use the Azure Open AI service, you must fill in the endpoint and API key of Azure OpenAI. The format of the endpoint is as follows:

https://{XXXXXXXX}.openai.azure.com/openai/deployments/{MODEL_NAME}/chat/completions?api-version={API_VERSION}

If you are instead using an OpenAI-compatible service (such as a self-hosted ollama API), you supply the endpoint and API key of that service.

Models

You can specify which model to use, if not specified, the default is gpt-3.5-turbo.

How to use it

Install the extension

To use it, first install the extension in your Azure DevOps organization. Click on the "Get it free" button and follow the prompts to install it. You may need to authorize the extension to access your Azure DevOps account.

Add the task to the build pipeline

After installing the extension, add the task to your build pipeline. Go to your build pipeline, click on the "+" icon to add a new task, and search for "Review PullRequest by GPT". Select it and add it to your pipeline.

Configure the task

Once you have added the task to your pipeline, configure it. In the task configuration, provide your API key for OpenAI API. To create an API key, go to https://platform.openai.com/account/api-keys.

Review Pull Requests

When the build is triggered from a Pull Request, the task will review it. If there is feedback on the changed code, the task will add comments to the Pull Request. If the build is triggered manually, the task will be skipped.

Compatible with Linux Build Agents

The tasks can execute on all supported build agent operating systems including Linux and MacOS.