🤖 CodeJury

Multi-Agent Code Review Pipeline — Run security, performance, style, and credential-tracing AI reviewers in parallel against GitHub PRs using the GitHub Copilot CLI, then synthesize findings into a ranked report. Exposed as an MCP tool so it can be invoked by Copilot, VS Code, or any MCP-compatible client.

Quick Start

# Install
npm install

# Build
npm run build

# Run as MCP server
node dist/index.js

# Run eval harness against real PRs (needs GITHUB_TOKEN)
export GITHUB_TOKEN=ghp_xxx
npm run eval

# Run tests
npm test

MCP Integration

CodeJury exposes a review_pr MCP tool via stdio transport — compatible with GitHub Copilot, VS Code, and any MCP client.

Copilot / VS Code

Add to your .github/copilot-instructions.md or MCP configuration:

{
  "mcpServers": {
    "codejury": {
      "command": "node",
      "args": ["C:/Workspace/CodeJury/dist/index.js"]
    }
  }
}

Then use the review_pr tool:

review_pr(url: "https://github.com/owner/repo/pull/42", dry_run: true)

Parameters: | Param | Type | Default | Description | |----------|---------|---------|-------------| | url | string | required | GitHub PR URL | | dry_run | boolean | true | If false, posts report as a PR comment |

How It Works

PR URL → fetch diff + file context
           │
     ┌─────▼────── Pre-Scan ───────────────────────┐
     │  16 grep patterns scan ALL changed files:    │
     │  • Credentials stored plaintext?             │
     │  • URLs logged without redaction?            │
     │  • Weak ciphers? Hardcoded keys?             │
     │  • DataStore/SharedPreferences without hash? │
     │  → produces priority score per file          │
     │  → auto-findings for critical matches        │
     └──────────────────────────────────────────────┘
           │
     ┌─────▼────── 4 Parallel Agents ────────────────────────────────────────┐
     │                                                                        │
     │  🔒 Security Agent                    ⚡ Performance Agent             │
     │  • Injection, secrets, auth           • N+1 queries, async blocking    │
     │  • Unsafe deserialization             • Unnecessary allocations        │
     │  • Input validation                   • Resource leaks                 │
     │                                                                        │
     │  🎨 Style Agent                       🔐 Credential Agent (v1.1)       │
     │  • Naming, duplication                • Traces credential lifecycle    │
     │  • Function length, complexity        • Entry → storage → log → cleanup│
     │  • Missing docs, magic numbers        • Reviews ONLY pre-scan-flagged  │
     │                                                                        │
     └────────────────────────────────────────────────────────────────────────┘
           │ (each runs with 20s timeout, retries once on schema failure)
     ┌─────▼────── Synthesis ───────────────┐
     │  • Deduplicate by file:line           │
     │  • Merge overlapping findings         │
     │  • Rank by severity (critical first)  │
     │  • Output Markdown + JSON             │
     └───────────────────────────────────────┘

Reviewer Agents

Each agent runs as an independent GitHub Copilot CLI subprocess (gh copilot suggest) with its own persona prompt. The orchestrator runs all agents in parallel via Promise.all, with per-agent timeouts and schema validation retries.

Each agent runs as an independent subprocess with its own system prompt. Agents that timeout or fail are excluded from synthesis — the pipeline never blocks on a single agent.

Configuration

Edit config/default.json or override with environment variables:

{
  "agents": {
    "timeoutMs": 20000,      // Per-agent timeout (NFR2)
    "maxRetries": 1,         // Schema validation retries (FR8)
    "copilotCliPath": "gh",
    "copilotCliArgs": ["copilot", "suggest"]
  },
  "github": {
    "token": "",             // Or set GITHUB_TOKEN env var
    "contextWindowLines": 50
  }
}

Set GITHUB_TOKEN in your environment for PR fetching and comment posting. Create a token at https://github.com/settings/tokens (only needs public_repo scope).

Eval Harness

The eval harness tests CodeJury against a fixed set of real PRs with annotated ground truth:

npm run eval

Dataset: eval/dataset.json — 7 annotated test PRs from well-known repos (Node.js, React, axios, lodash, etc.)

Metrics computed:

• Precision, recall, F1 score per agent
• Overall precision/recall/F1
• Results saved to .eval-results/ as timestamped JSON

To add new test cases, edit eval/dataset.json with a PR URL and expected findings.

Project Structure

CodeJury/
├── src/
│   ├── index.ts              — Entry point
│   ├── config/index.ts       — Config loader (.json + env)
│   ├── types/index.ts        — Zod schemas + TypeScript types
│   ├── github/client.ts      — GitHub API: fetch PR diff + file context
│   ├── scanner/
│   │   └── preScan.ts        — 16-pattern grep phase (credential, crypto, logging)
│   ├── agents/
│   │   ├── base.ts           — Abstract agent: subprocess, parse, retry
│   │   ├── reviewers.ts      — Security, Performance, Style agents
│   │   └── credentialAgent.ts — Credential-tracing agent (pre-scan aware)
│   ├── orchestrator/
│   │   └── pipeline.ts       — Parallel execution, synthesis, dedup
│   ├── mcp/server.ts         — MCP server (stdio transport)
│   └── utils/                — Logger, metrics, helpers
├── eval/                     — Eval harness
│   ├── dataset.json          — 7 annotated test PRs
│   ├── scorer.ts             — Precision/recall/F1
│   ├── reporter.ts           — Summaries + output
│   └── run.ts                — End-to-end eval runner
├── tests/                    — 32 unit tests (5 suites)
├── config/default.json       — Default configuration
├── docker-compose.yml        — One-command Docker setup
└── package.json

API

review_pr (MCP Tool)

Runs the full pipeline against a GitHub PR.

// Input
{ url: string, dry_run?: boolean }

// Output
{
  pr_url: string,
  summary: string,          // "Found 8 issue(s): 1 critical, 3 high, 4 low..."
  findings: [{
    file: string,
    line: number | null,
    severity: "critical" | "high" | "medium" | "low",
    issue: string,
    suggestion: string,
    agent: "security" | "performance" | "style"
  }],
  agent_status: {
    security: "ok" | "timeout" | "error",
    performance: "ok" | "timeout" | "error",
    style: "ok" | "timeout" | "error"
  },
  latency_ms: number
}

Docker

docker compose up --build

Requires GITHUB_TOKEN environment variable.

Requirements Coverage

See Requirements.md for the full specification. CodeJury satisfies all 20 functional requirements (FR1–FR20) and 12 non-functional requirements (NFR1–NFR12).

License

MIT