Azure DevOps Extension for Automated Code Reviews with Azure OpenAI
Overview
This project is an Azure DevOps Extension that automates code reviews using Azure OpenAI. Once a pull request (PR) is created in a repository, this extension triggers a code review process powered by Azure OpenAI, providing comprehensive, actionable feedback directly in the PR.
Key Features
- Principal Engineer-Level Reviews: Leverages advanced prompts with expertise in architecture, security, performance, and modern development practices
- Latest OpenAI Models: Built on the newest OpenAI package supporting GPT-4 and beyond
- Comprehensive Coverage: Reviews up to 10 issues per severity level (Critical, Important, Suggestions), adaptive to PR size
- Enhanced Reliability: Built-in retry logic with exponential backoff for robust operation
- Existing Comment Integration: Analyzes and tracks resolution status of previous PR comments
- Modern Best Practices: Covers security vulnerabilities, performance optimization, SOLID principles, async/await patterns, and more
- Azure DevOps Optimized: Clean file references (no complex hunk notation) for easy navigation
- Actionable Feedback: Clear before/after code examples with specific impact explanations
- Customizable Review Templates: Configure focus areas, severity levels, and custom instructions
- Seamless Integration: Works directly within your Azure DevOps PR workflow
Task Details
This extension includes a custom task called Pull Request Code Review with OpenAI, which automates the code review process for pull requests using Azure OpenAI.
- ID:
5ac5fe12-459c-47e7-a278-26bd07f7f4e3
- Name:
AIPRCodeReview
- Friendly Name:
Pull Request Code Review with OpenAI
- Description: Automatically reviews code changes in pull requests using OpenAI.
- Category:
Utility
- Author:
GN Group
- Visibility:
Build
- Minimum Agent Version:
1.95.0
- Execution: Compatible with Node 10 and Node 16.
| Input Name |
Type |
Required |
Default Value |
Description |
api_key |
string |
Yes |
|
The API key for accessing OpenAI or Azure OpenAI. |
ai_model |
string |
No |
gpt-4o |
The OpenAI model to use for code review (e.g., gpt-4o, gpt-4-turbo). |
file_extensions |
string |
No |
|
Comma-separated list of file extensions to include in the review (e.g., .cs,.ts,.js). |
base_api_url |
string |
Yes |
|
Base API URL to OpenAI service (Azure OpenAI endpoint or OpenAI API). |
file_excludes |
string |
No |
|
Comma-separated list of file patterns to exclude (e.g., sitecore/,/*.json). |
disable_default_prompts |
boolean |
No |
false |
Set to true to use only custom prompts, disabling the default principal engineer prompts. |
additional_prompts |
string |
No |
|
Additional custom instructions to include in the code review prompt. |
include_existing_comments |
boolean |
No |
false |
Include existing PR comments in the review to track resolution status. |
review_severity |
string |
No |
Standard |
Review depth: Critical (blocking only), Standard (comprehensive), Comprehensive (all issues). |
focus_areas |
string |
No |
|
Comma-separated priority focus areas (e.g., security,performance,testing). |
bugs |
boolean |
No |
true |
Check for bugs and logic errors in the code. |
security |
boolean |
No |
true |
Flag potential security vulnerabilities (SQL injection, XSS, auth issues, etc.). |
naming_conventions |
boolean |
No |
true |
Flag naming convention and code style issues. |
error_handling |
boolean |
No |
true |
Flag missing or improper error handling. |
performance |
boolean |
No |
true |
Flag performance issues (N+1 queries, memory leaks, inefficient algorithms). |
best_practices |
boolean |
No |
true |
Provide modern best practice recommendations (async/await, LINQ, patterns, etc.). |
reset_comments |
boolean |
No |
false |
Reset all comments in the pull request before adding new review comments. |
max_tokens |
number |
No |
4000 |
Maximum tokens for OpenAI response (increased for comprehensive reviews). |
retry_attempts |
number |
No |
3 |
Number of retry attempts for failed API calls with exponential backoff. |
Review Output
The extension provides structured feedback in three severity levels:
- 🚨 Critical Issues (Up to 10 blocking issues): Security vulnerabilities, critical bugs, breaking changes, performance blockers, production risks
- ⚠️ Important Issues (Up to 10 significant concerns): Security concerns, bugs, performance issues, architecture violations, error handling gaps
- 💡 Suggestions (Up to 10 optional improvements): Code quality, modern patterns, best practices, maintainability enhancements
Each issue includes:
- Clear file location (e.g.,
ProductRepository.cs, line 42)
- Specific problem description with technical details
- Impact explanation (why it matters)
- Concrete fix with before/after code examples
- Metadata for Azure DevOps integration
Execution
This task runs on the Node.js environment and supports both Node 10 and Node 16. It triggers when a pull request is created or updated and provides comprehensive feedback directly on the PR using Azure OpenAI's latest models with built-in retry logic for reliability.
Messages
- TaskCompleted: "The pull request has been successfully reviewed."
- SkippingTask: "This task is skipped because it's not triggered by a Pull Request."
- FailedOAuth: "'Allow Scripts to Access OAuth Token' must be enabled for this task."
Prerequisites
Before using the Pull Request Code Review with OpenAI task, make sure that your Azure DevOps build service has the necessary permissions to contribute to pull requests in your repository.
Step 1: Give Permission to the Build Service Agent
- Navigate to your repository settings in Azure DevOps.
- Select Security from the left menu.
- Under Permissions, find the build service user (usually in the format
[ProjectName] Build Service ([OrganizationName])).
- Ensure that the Contribute to pull requests permission is set to Allow.
This permission is necessary for the task to add review comments directly to your pull requests.
Installation and Setup
To install this Azure DevOps extension:
- Navigate to your Azure DevOps project.
- Go to the Extensions Marketplace and search for "Azure DevOps Extension for Automated Code Reviews with Azure OpenAI".
- Click on the extension and follow the installation steps.
Configuration
Add the Task to Your Pipeline:
- Edit your pipeline YAML file or use the classic editor.
- Add the Pull Request Code Review with OpenAI task to the pipeline.
Provide the Required Inputs:
- Make sure to enter your OpenAI API Key and configure the review options according to your needs.
Set Up API Authentication:
- Ensure that your API key has the appropriate permissions to access OpenAI services.
Usage
Once the extension is installed and configured, it will automatically trigger a comprehensive code review every time a pull request is created or updated. The review provides:
- Comprehensive Analysis: Up to 10 issues per severity level, covering all significant concerns
- Modern Best Practices: Security, performance, architecture, async patterns, LINQ usage, and more
- Clear Actionable Feedback: Specific file locations, impact explanations, and before/after code examples
- Comment Tracking: Optional analysis of existing PR comments to track resolution status
The results are added as structured comments directly on the PR, helping your team identify and fix issues faster with professional, principal engineer-level insights.
Basic Configuration Example
- task: AIPRCodeReview@3
inputs:
base_api_url: "$(OpenAI_BaseUrl)"
api_key: "$(OpenAI_ApiKey)"
ai_model: "gpt-4o"
file_extensions: ".cs,.js,.ts,.tsx"
file_excludes: "sitecore/**,**/*.json"
max_tokens: 4000
Advanced Configuration with Custom Focus
- task: AIPRCodeReview@3
inputs:
base_api_url: "$(OpenAI_BaseUrl)"
api_key: "$(OpenAI_ApiKey)"
ai_model: "gpt-4o"
file_extensions: ".cs,.ts"
file_excludes: "Tests/**,Migrations/**"
review_severity: "Comprehensive"
focus_areas: "security,performance,async patterns"
include_existing_comments: true
additional_prompts: "Pay special attention to database queries and Entity Framework usage"
max_tokens: 4000
retry_attempts: 3
What's New in v3.0
- ✨ Latest OpenAI Package: Support for GPT-4o and newest models
- 🔄 Retry Logic: Exponential backoff for improved reliability
- 🎯 Enhanced Prompts: Principal engineer persona with comprehensive modern best practices
- 📊 Expanded Coverage: Up to 10 issues per severity level (adaptive to PR size)
- 📍 Better Formatting: Clean file references without complex hunk notation
- 💬 Comment Integration: Track resolution status of existing PR comments
- 🛡️ Security: Prompt injection prevention for untrusted comment content
Contributions
We welcome contributions! Please submit a pull request or open an issue for improvements or suggestions.
| |
