ArcLight Claude PR Review

Automated pull request code review for Azure DevOps, powered by the Claude Code CLI. The task is a dependency-free Node 24 program. The review runs on every PR (typically as a Build Validation policy), reads the PR diff and linked Azure Boards work items, and posts findings directly as PR comment threads.

Maintained by ArcLight. Repository: viettdann/arclight-ado-claude-pr-review. Published on the Marketplace as ArclightVn.arclight-claude-pr-review.

git clone https://github.com/viettdann/arclight-ado-claude-pr-review.git

How it works

1. The pipeline checks out the PR with full history and runs runner.js.
2. The task pulls PR details, iteration changes, and (optionally) linked work items through the Azure DevOps REST API.
3. It renders the prompt from prompt.txt for this run's mode and Minor-findings cap, appends your custom prompt as an Additional Direction section if you set one, and appends the administrator-configured Review Behavior Directives. Where instructions conflict, the later one wins, so the directives outrank your direction, which outranks the template.
4. It runs claude -p with --output-format stream-json, streaming tool calls and text deltas to the pipeline log. The agent posts review comments by invoking add-claude-comment.mjs / update-claude-comment.mjs, which the task stages into the working directory for it.
5. Findings appear as PR threads tagged _[Generated by ArcLight Claude Code Review]_, headed by one summary comment; on re-review, threads whose issues were addressed get resolved.

Every review posts one summary comment and rewrites it in place on later rounds. Besides the table of open findings, it records the iteration and commit the round reviewed, which is what the next round diffs against to tell new changes from old ones; with no summary to read, the previous iteration's commit from Iteration_Details.txt serves instead.

On re-review the agent accounts for each of its own open threads exactly once: it resolves the ones the push fixed, replies on the ones the push moved but did not fix, and stays silent on the ones nothing touched. It never reopens a thread someone closed, even when the defect is still there; a Critical or Major finding closed while still live is recorded on the summary comment instead, so the decision stays traceable without the thread being argued with. The summary and the Minor roll-up are the two exceptions: they belong to the review, so it sets their status every round.

runner.js is the task entrypoint; inputs reach it as the INPUT_<NAME> environment variables the agent sets for task inputs.

Usage

The ArclightClaudePrReviewV1 task runs on the agent's Node24 handler, which first shipped in agent 4.265.1. That is the manifest's minimumAgentVersion, so an older agent is upgraded before the task runs. Microsoft-hosted agents are always current. Once the extension is installed in the organization:

# No pr: block, Azure Repos ignores it. A branch policy runs this, see below.
trigger: none

pool:
  vmImage: ubuntu-latest

steps:
- checkout: self
  # Required unless diffOnlyReview: the agent explores the checked-out repo.
  fetchDepth: 0

- task: ArclightClaudePrReview@1
  displayName: 'Claude Code PR Review'
  inputs:
    # Exactly one of these two. Keep it in a secret variable group.
    claudeCodeOAuthToken: $(CLAUDE_CODE_OAUTH_TOKEN)
    #anthropicApiKey: $(ANTHROPIC_API_KEY)

    # Optional, defaults shown. Org/project/repo/PR id are auto-derived.
    # model: ''                       # empty = CLI default
    # reviewDepth: 'standard'         # quick | standard | deep
    # timeout: '15'                   # minutes, raise it for deep
    # prompt: ''                      # extra direction, see below
    # diffOnlyReview: false           # cheaper on large repos, not for deep
    # maxMinorIssues: '5'             # 0 suppresses Minor findings
    # suppressPositiveFeedback: false
  env:
    # Required, Azure DevOps does not pass this to a task automatically.
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)

Full list under Inputs. Secrets flow through inputs and env: only, never macro-expanded into a script body.

Running it on pull requests

Azure Repos has no PR trigger. A pr: block is parsed and then ignored, unlike GitHub. What runs the pipeline on a PR is a Build Validation branch policy on the target branch, so the pipeline alone does nothing until you add one.

Repos → Branches → main → ⋯ → Branch policies → Build validation → +:

One policy per target branch: a policy on main covers PRs into main only. With Automatic, the review runs when the PR opens and again on every push to the source branch.

Prerequisites

The task targets Microsoft-hosted agents. ubuntu-latest supplies everything it needs: the Node24 task handler, npm for installing the Claude Code CLI when the image does not already carry it, git for diffOnlyReview, and outbound access to api.anthropic.com.

Self-hosted agents are out of scope and untested. One limitation worth naming if you try anyway: the task talks to both APIs through Node's built-in fetch, which ignores HTTP_PROXY and HTTPS_PROXY, so an agent that reaches the internet only through a proxy will not work.

Authentication

Provide exactly one of:

Azure DevOps access is always the pipeline's System.AccessToken. Map it into the step with env: SYSTEM_ACCESSTOKEN: $(System.AccessToken), since the agent does not expose it automatically, and grant the Build Service identity Contribute to pull requests on the repository. There is no PAT input; Azure DevOps Server is not supported.

Inputs

Prompt inputs are mutually exclusive, setting more than one fails the run. promptRaw/promptFileRaw bypass all rewriting; prompt/promptFile are appended to the rendered prompt.txt as an Additional Direction section, untouched, and followed by the directives block; with none set, the rendered prompt.txt and the directives are used alone.

Review depth

The prompt's triage phase walks six scan angles: A line-by-line, B removed behavior, C cross-file impact, D requirements coverage against linked work items, E reuse/simplification/efficiency, and F tests and conventions. reviewDepth scales that set.

Depth is applied as a directive appended to the prompt, so the prompt is the same size at every tier. The cost difference is in how much the agent explores: turns, tool calls, subagents, and output. It is not a difference in input tokens; to cut those, use diffOnlyReview.

deep fans out to subagents and verifies each candidate, so it needs headroom: raise timeout and maxTurns before enabling it, and expect a larger bill. It posts nothing until triage finishes, so a run that hits the timeout loses the whole review rather than landing partial findings; the task warns when deep is combined with a timeout at or below the 15-minute default. It also assumes a checked-out repo, since angles B, C and F look outside the diff; combining it with diffOnlyReview warns and quietly narrows the review.

reviewDepth is independent of reasoningEffort. Depth changes the review procedure in the prompt; reasoningEffort tunes how hard the model thinks at each step. They compose.

Tuning out review noise

The prompt ships a Do not report list covering the usual noise: lock and generated files, out-of-date dependencies, formatting, missing tests on mechanical changes, unreachable defensive checks, theoretical races. To extend it with rules of your own, pass them through prompt or promptFile; they are appended as an Additional Direction section that overrides the built-in list. Writing a rule down once is cheaper than arguing with the same comment every round:

- task: ArclightClaudePrReview@1
  inputs:
    anthropicApiKey: $(ANTHROPIC_API_KEY)
    prompt: |
      Known non-issues in this repository, do not report them:
      - Files under src/Generated/ are scaffolded from the OpenAPI spec.
      - EF Core migrations are generated; review the model change, not the migration.
      - We deliberately do not use constructor DI in the Legacy namespace.

One-time organization setup

1. Grant the project Build Service identity Contribute to pull requests (and Read) on each target repository: Repos → Manage repositories → Security.
2. Install the extension from Organization Settings → Extensions → Browse private marketplace.
3. Store ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN in a secret variable group shared across app repos.
4. Add the pipeline as a Build Validation branch policy on each protected branch, per Running it on pull requests above. Nothing runs without this.
5. Under Project Settings → Pipelines → Settings, enable Protect access to repositories in YAML pipelines, then set the token scope per Token scope and work items below. The review agent runs with the full Claude Code toolset and System.AccessToken in its environment while reading untrusted PR content, so its reach should be set deliberately rather than left at whatever the org default happens to be.

Security model

Treat the PR title, description, diff, and linked work items as attacker-controlled input that is fed to an agent running with --dangerously-skip-permissions. The permissions granted to the Build Service identity are the security boundary, and nothing else meaningfully constrains the agent.

In particular, the CLI's --disallowedTools Bash(git push *) rule is a speed bump, not a control: the same capability is reachable through the Azure DevOps REST API with the token that is already in the environment. Do not size the token's permissions on the assumption that the deny rule holds. Keep Protect access to repositories in YAML pipelines on, scope the token to the minimum the review needs, and prefer read-only grants wherever the review only reads.

The task manifest restricts logging commands (restrictions.commands.mode: restricted, settableVariables.allowed: []) and every line of agent-influenced text is passed through a sanitiser that breaks ##vso[ and ##[ markers before logging, so agent output cannot forge pipeline logging commands. Those are defence in depth; they do not change the boundary above.

Token scope and work items

project locates the repository and the pull request. Linked work items are fetched by id from the collection, so a pull request may link items tracked on a board in any project the pipeline token can read.

The token's reach is what has to be configured:

The second row is a real widening, so weigh it rather than copying it: the job then runs as the collection-scoped Build Service identity, and that applies to every pipeline in the project, not only this one. Two things limit the damage: Protect access to repositories in YAML pipelines governs repository reach independently and should stay on either way, and the review only ever reads work items, so the grant on the boards project can be read-only.

Unreachable ids never fail the run: readable items still come back, and the rest are logged as WORK_ITEMS_UNREADABLE: <ids>. Grep the pipeline log for that string when a review looks like it is missing context: it means the token could not read those items, so the grant above is incomplete.

License

Copyright (C) 2026 ArcLight.

Licensed under GPL-3.0-or-later, see LICENSE.