AI Code Reviewer for Azure DevOps

Automated AI code review on every pull request — directly in Azure DevOps, using your own Azure OpenAI. No extra servers, no data leaving your tenant.

What makes it different: it learns your team's own standards from past PR comments and enforces them automatically on new PRs. Not generic best practices from the internet — your rules, your history.

How it works

1. Install the extension, enter your Azure OpenAI endpoint in Project Settings → AI Code Reviewer.
2. Add one pipeline task to your PR validation pipeline:

   - task: PrReviewLearning@1
     inputs:
       mode: review
   

3. Every new PR gets reviewed automatically. AI comments appear inline, like a human reviewer.
4. When your team resolves a comment and pushes a fix, the extension learns that rule and applies it to all future PRs across your repos.

Features

• Inline comments on every PR — findings appear directly on the changed lines
• Learns from your history — rules extracted from resolved PR comments, not internet opinions
• #best-practice tag — mark any comment to immediately create a rule, no AI classification needed
• Project groups — one rule can cover a set of related repositories
• Three review modes — full AI, rule-based only, or disabled — choose per pipeline
• Four learning modes — strict (default), tagged-only, conversation, all
• Rule approval workflow — non-tagged rules require admin approval before affecting PRs
• Rule provenance & audit log — every rule shows its source PR comments and full change history
• Feedback-loop protection — ingest always skips AI-posted comments (never learns from itself)
• API key encrypted at rest — AES-GCM with per-organization key before writing to Extension Data Service
• Your data stays in your tenant — BYOK Azure OpenAI, stored in ADO Extension Data Service
• Free — pay only for your own Azure OpenAI token usage (~$5–50/month typical)

Requirements

• Azure DevOps (cloud or on-prem)
• Azure OpenAI resource with a chat model (e.g. gpt-4o-mini) and embedding model (text-embedding-3-small)
• Azure Pipelines

Privacy & security

Source code, PR diffs, and comments never leave your Azure tenant. All AI calls go directly to your own Azure OpenAI deployment. AISRE Labs never sees your code.

Full privacy policy

Pipeline setup

1. Review pipeline — runs on every PR

Save as ai-review-pipeline.yml in your repo root, then register it as a Branch Policy → Build Validation on your target branch (e.g. main). Set it as Optional to give feedback without blocking merge:

trigger: none

pr:
  branches:
    include: [main]
  paths:
    exclude:
      - '**/*.md'
      - docs/*

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: Review
    timeoutInMinutes: 5
    steps:
      - task: PrReviewLearning@1
        inputs:
          mode: review

2. Ingest pipeline — learns rules from a merged PR

Save as ai-ingest-pipeline.yml. Run manually after merging a PR, passing the PR number as a parameter. The plugin will analyze that PR's resolved comments and create/update learned rules:

trigger: none

parameters:
  - name: pullRequestId
    displayName: 'Pull Request ID to ingest'
    type: string
    default: ''

pool:
  vmImage: 'ubuntu-latest'

jobs:
  - job: IngestRules
    timeoutInMinutes: 10
    steps:
      - bash: |
          echo "##vso[task.setvariable variable=System.PullRequest.PullRequestId]${{ parameters.pullRequestId }}"
        condition: ne('${{ parameters.pullRequestId }}', '')

      - task: PrReviewLearning@1
        inputs:
          mode: ingest

Typical workflow: reviewer asks for a change in a PR, you fix it and merge, then run this pipeline with the PR number → the rule is captured for all future PRs across the project.

Task inputs reference

How to register the pipelines in Azure DevOps

After saving the YAML files in your repo, each one needs to be registered as a pipeline in ADO:

1. Pipelines → New pipeline
2. Azure Repos Git → pick your repo
3. Existing Azure Pipelines YAML file → select /ai-review-pipeline.yml (or /ai-ingest-pipeline.yml)
4. Save (don't run yet)
5. Optionally rename the pipeline (three-dot menu → Rename), e.g. myrepo-ai-review and myrepo-ai-ingest

Set review pipeline as Branch Policy (one-time)

So the review runs on every PR targeting main:

1. Project Settings → Repositories → [your repo] → Policies
2. Under Branch Policies, click on main (or your target branch)
3. Build Validation → Add
4. Build pipeline: select myrepo-ai-review
5. Trigger: Automatic
6. Policy requirement: Optional (recommended — AI review is advisory, not blocking). Pick Required only once you trust its precision.
7. Save

When does each mode actually run?

Review — automatic, every PR

Once the Branch Policy is set up:

• Dev opens a PR targeting the protected branch → review pipeline triggers automatically
• AI analyzes the diff, applies learned rules, posts inline comments on the PR
• Dev pushes new commits → pipeline re-runs; already-posted comments are not duplicated (idempotent by file+line+rule hash)
• Nothing manual — set it once, forget it.

Ingest — manual, after merge

The ingest pipeline requires a pullRequestId parameter, so it's run on demand:

1. Reviewer leaves a comment on a PR (optionally tagged #best-practice to skip AI classification and make it a rule immediately)
2. Author fixes and merges the PR → note the PR number (e.g. #42)
3. In ADO: Pipelines → myrepo-ai-ingest → Run pipeline
4. In the "Run pipeline" dialog, paste the PR number into Pull Request ID to ingest
5. Run
6. Plugin analyzes the PR's resolved comments and captures them as rules; from the next PR onwards, those rules apply automatically

Not every merge needs an ingest run — only the ones where a reviewer caught something worth remembering as a rule.

Project Groups — scope rules across related repos

By default, a rule learned in one repository applies only to that repository. If your team works across multiple repos that share conventions, you can group them so a rule learned in one applies to all.

When do you need it?

• You have microservices split across multiple repos that share coding standards (payment-api, payment-worker, payment-gateway)
• A reviewer notes "we always use HikariCP for DB pooling" in one of them — without project groups, this rule applies only to that one repo. With a payments group, it applies to all three.
• Not needed if you have one repo or each repo has different standards.

How to configure

1. In ADO: Project Settings → AI Code Reviewer → Project groups
2. + New group
3. Fill in:
   • Name — e.g. payment-services
   • Description (optional)
   • Repositories (one per line, format org/project/repo):

     myorg/MyProject/payment-api
     myorg/MyProject/payment-worker
     myorg/MyProject/payment-gateway
     

4. Save

From the next ingest onwards, the AI classifier sees your groups and can scope a rule to a group instead of just one repo. Add a new repo to the group later → all group-scoped rules apply to it automatically, no re-ingest needed.

⚠ Pipelines are still per-repo

A project group shares rules across repos, not pipelines. Each repo in the group still needs:

• Its own ai-review-pipeline.yml in the repo root
• Its own pipeline registration in ADO (one-time, via New pipeline wizard)
• Its own Branch Policy → Build Validation set on main

Same for ingest — each repo needs its own ai-ingest-pipeline.yml to run ingest on a merged PR from that repo. The group controls how rules match; the pipelines control where reviews run.

A practical setup for a 3-repo group: 6 pipelines total (review + ingest × 3 repos), all sharing the same learned rules store via the extension's per-organization data service.

Scope hierarchy

The AI picks the narrowest scope that fits each rule:

You can edit any rule's scope later in the Learned Rules tab if AI classified it too narrow or too broad.

Enterprise governance — learning mode, provenance, audit log

Learning mode — control which comments become rules

Set in Azure OpenAI tab → Learning mode section. Four strictness levels:

Rule approval workflow (non-tagged rules only)

Rules created from comments explicitly tagged #best-practice or #best-global-practice are pre-approved — the reviewer intentionally opted in by using the tag, so they go live immediately.

Non-tagged rules (those inferred from Resolved + commit-touched-line heuristic) are marked pendingApproval: true when created. They are NOT applied to any PR review until an admin explicitly approves them in the Learned Rules tab.

• Open Hub → Learned Rules → filter "Only pending approval"
• Click Approve on rules that make sense, Reject to discard
• Every approval/rejection is recorded in the audit log

This prevents the plugin from silently enforcing rules the team didn't explicitly agree on — especially important under conversation or all learning modes where AI interpretation is involved.

Feedback-loop protection

Ingest always skips comments posted by the AI review bot itself — regardless of learning mode. This is a hard-coded safety to prevent the plugin learning from its own output (e.g., AI posts a rule violation, dev disagrees → AI re-learns the violation as a new rule). The detection checks bot prefix, known service-account authors, and rule-violation comment structure.

Why strict is the default

The old "all" behavior could turn ambiguous PR discussions into enforced rules — a reviewer saying "hmm, this looks weird" followed by any commit touching that line would become a project-wide rule, with AI extrapolating the meaning. Strict mode closes this by requiring both explicit human signals:

1. The thread is marked Resolved in the PR UI (reviewer verified the fix)
2. A commit actually changed the commented line (there's evidence of action)

Plus, non-tagged rules are forced to PROJECT scope — they can't accidentally become cross-repo policies. To create a global rule intentionally, reviewers use the explicit #best-global-practice tag.

Rule provenance — click Details to see source PRs

Every rule in the Learned Rules tab has a Details button. Expanding it shows:

• Provenance — the exact PR comments that created or reinforced this rule, with reviewer name, date, file path / line, and deep-link back to the PR
• Audit log — chronological history of every change: creation, ingest merges, confidence adjustments, activation toggles, manual edits, deletions

Audit log — who changed what and when

Every state change on a rule is recorded. Each event captures:

• Actor — system:ingest (PR #42) for automated events or user:<display name> for Hub UI changes
• Action — created / merged / confidence_changed / active_toggled / edited / deleted
• Before/after — only the fields that changed (compact audit footprint)
• Human-readable note — e.g. "confidence 0.50 → 0.70" or "conflict accepted (rule kept active)"

Audit events are sharded (~100 per document) in the same per-organization Extension Data Service as the rules themselves — zero external dependencies, zero new infrastructure.

Support

Issues and feedback: github.com/aisre-labs/pr-review/issues