Agent Diff Auditor

Review what your AI agent actually changed. It flags skipped or deleted tests, weakened assertions, suppressed errors and softened CI in your git diff, so a "done" that is not really done stands out. It works with any agent (Copilot, Claude Code, Cursor, Codex...) and needs no API key. Every rule is plain pattern matching, so it is free and fast.

The recording is a sample repository where an agent skipped a test, weakened an assertion, added a @ts-ignore, swallowed an error and softened CI. Diff Auditor lists each one with the file and line, and you can mark the ones you accept as reviewed.

What it flags

Untracked files are audited too, because agents often add new test files.

Use it

In VS Code: run Diff Auditor: Audit Changes. Findings show in the Diff Auditor view in the Explorer, in the Problems panel and in the status bar. Click one to jump to the line.

Hiding findings you have accepted. Set Baseline (Hide Current Findings) hides everything that exists now, so only what an agent adds later is reported. To accept a single finding, click the check mark next to it in the view, or use the Diff Auditor: mark as reviewed quick fix (lightbulb) on the line. Reviewed findings stay hidden when their line moves, but a second copy of the same shortcut is reported. They are stored in .git/diffaudit-baseline.json, which is never committed, and Clear Baseline brings everything back. The CLI and the Claude Code hooks use the same file. By the agent itself: the extension registers a language model tool, #auditChanges. In Copilot agent mode you can tell the agent to run it before it says it is done.

From a terminal, hook or CI:

npm run compile
node out/src/cli.js --fail-on high

Options: --base <ref>, --stdin, --no-untracked, --ignore <rule-id>, --json, --sarif, --output <file>, --min-severity, --fail-on high|medium|low|never. Exit code is 1 when a finding reaches --fail-on, so it can gate a commit, an agent hook or a CI job.

As a pre-commit hook. Put this in .git/hooks/pre-commit (and make it executable) to stop a commit that contains a high-severity finding. A hook sees the working tree, so it checks what you are about to commit plus anything unstaged; use --base or the baseline if that is too broad:

#!/bin/sh
node /path/to/AIDiffAuditor/out/src/cli.js --fail-on high || {
  echo "Diff Auditor found high-severity shortcuts. Review them, or commit with --no-verify." >&2
  exit 1
}

Comparing with a branch. diffaudit --base origin/main (or the Diff Auditor: Audit Against Branch or Commit... command, or baseRef in .diffauditor.json) reports what you changed since you branched off, even if the branch has moved on since. It diffs against the merge base, not the branch tip, so a test a teammate added to main after you branched is not reported as deleted. An unknown ref is an error that says so.

In CI with GitHub code scanning. --sarif writes a SARIF 2.1.0 report that GitHub shows as annotations on the pull request. --output <file> writes it to a file, so the exit code can still gate the job:

name: Diff audit
on: pull_request
permissions:
  contents: read
  security-events: write
jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - name: Build the auditor
        run: |
          git clone --depth 1 https://github.com/zhongwu/AIDiffAuditor.git "$RUNNER_TEMP/diffaudit"
          (cd "$RUNNER_TEMP/diffaudit" && npm ci && npm run compile)
      - name: Audit the pull request
        run: node "$RUNNER_TEMP/diffaudit/out/src/cli.js" --base "origin/${{ github.base_ref }}" --sarif --output diffaudit.sarif --fail-on never
      - uses: github/codeql-action/upload-sarif@v3
        with: { sarif_file: diffaudit.sarif }

Use --fail-on high instead of never to fail the job on high-severity findings (upload the report in an if: always() step so it still appears). Severity maps to the SARIF level: high is error, medium warning, low note. Each result carries a fingerprint based on rule, file and code (not the line number), so GitHub recognises the same finding after it moves. Code scanning on private repositories needs GitHub Advanced Security. The workflow above has not been run on GitHub yet; the report itself is checked against the official SARIF schema.

Settings

diffAuditor.baseRef (default HEAD), diffAuditor.includeUntracked, diffAuditor.minSeverity, diffAuditor.ignoreRules, diffAuditor.autoAudit (re-audit on save).

Team config

Commit a .diffauditor.json at the repository root so everyone, the editor, the CLI, CI and the agent hooks, uses the same defaults:

{
  "ignoreRules": ["retries-added"],
  "minSeverity": "medium",
  "ignorePaths": ["legacy/**", "**/*.generated.ts"],
  "baseRef": "origin/main"
}

Unknown keys and wrong types are errors with a message that names the problem, so a typo never silently turns a check off. VS Code validates and completes the file as you type.

Develop

npm install
npm test        # compiles, then runs the node:test suite

• src/core/ has no VS Code dependency: diffParser.ts (unified diff to numbered lines), rules.ts (all rules and their patterns), audit.ts (runs rules, filters, sorts).
• src/git.ts collects the diff, src/extension.ts is the VS Code shell, src/cli.ts is the CLI.
• To add a rule: write it in rules.ts, add it to RULES, add a test in test/rules.test.ts.

Not done yet

• It matches patterns in the diff; it cannot tell whether a test is meaningful.
• Tests deleted because the code moved to another repository are still reported. Mark them as reviewed.
• Heuristics can misfire, for example a legitimate // @ts-expect-error in a type test. Use diffAuditor.ignoreRules or .diffauditor.json to switch a rule off.
• Measured on human history and hand-written agent-style diffs, not yet on many real agent sessions.