IssueTriage

Intelligent AI Readiness for GitHub Issues

IssueTriage turns every GitHub backlog into an automation-ready funnel. Inside VS Code and Cursor you can triage issues, surface historical risk, run context-aware AI assessments, and decide which work can safely go to coding agents.

Why Use IssueTriage

• Prioritize by evidence – Composite readiness scores combine requirements clarity, code complexity, security sensitivity, and business impact.
• Catch hidden risk – Linked pull requests, change volume, and review friction automatically temper optimistic scores.
• Work where you code – Native sidebar views, commands, and status bar entry points for both VS Code and Cursor.
• Stay in control – All assessment and risk data is stored locally in SQLite; you decide whether to publish results back to GitHub.
• Deep GitHub integration: issues, labels, assignees, milestones, linked PRs, commit history.
• Flexible LLM routing (50 assessments free per month): Use our API AI key (limited to 50 issues a month) or direct OpenRouter calls with your own key.
• LLM Usage Tracking: Usage based tracking by UsageTap

Why IssueTrage was developed

• We wanted to quickly assess a long backlog of issues to find which ones were ready to develop using the AI coding tools. We set out to create the assessment model to determine when things are "AI development ready."
• "Plan Explosion." We used planning mode and created solid specification files, but after a couple of weeks there were dozens of these documents that will NEVER be read again. Also, these documents lacked the Question and Answers given along the way. We wanted a record of the decisions, and we wanted to know when there were no more implementation questions. It seemed logical to keep these with the GitHub issues as comments.
• Risk Model." We have an interest in assessing proposed issues risk and complexity. The data captured on prior issues and how well they were automated could help decide between what issues to do first. Low risk high or equal value first. We are just starting this journey, and you will see Risk Analysis and the ability to download a ML training data to train your own model.

Architecture at a Glance

Interface Tour

1. Full Assessment Panel

• Tabbed layout (Issues, ML Training, LLM Usage).
• Score breakdown, summary, recommendations, automation badge, assessment history timeline.
• Risk Intelligence section with top drivers and PR evidence.

2. Activity Bar Sidebar

• Repository selector, readiness distribution, grouped issue lists (Not Assessed, Ready, Prep Required, Needs Review, Manual Only).
• Inline quick actions: assess, refresh, send to automation.
• Color-coded badges that mirror assessment readiness.

3. Opportunity Mix Matrix

• Scatter plot of business value vs. readiness using assessed issues only.
• Hover to inspect, click to open GitHub, legend keyed to readiness states.

4. Assessment Question Flow

• Guided checklist for automation readiness questions, each with AI-assisted answers and follow-up prompts.

GitHub oAuth Authentication (first time used)

To use IssueTriage you need to authenticate with GitHub. To get connected:

1. Install the IssueTriage extension by clicking on the Extensions side bar or View - Extensions from the menu (VSCode & Cursor), searching for IssueTriage and clicking install
2. Click on the IssueTriage diamond icon in the sidebar (vscode)
3. Click on the Connect to Github button in the side panel or the extensions main panel
4. You will be redirected to GitHub for "Device Activation" - select the right account and click continue.
5. You will be asked for an 8 digit code. Go back to your VSCode or Cursor and there will be a notification in the footer (click on the notification bell very bottom right of your VSCode or Cursor) window. Copy the code and enter it in the GitHub device activation window
   
   
6. You will then be asked to authorize PredictabilityAtScale (thats us). REassess your life choices and click Agree/Yes. If you have 2FA (and we hope you do) you may be asked to further authenticate; Do the steps asked.

If successful you will see a "Congratulations" message and you can return to VSCode or Cursor.

NOTE: You may need to click Refresh or choose a different repository by clicking on the repo name and selecting from the drop-down of all repos listed. We try and choose the repo matching your open folder repo. If no issues are showing just click the refresh button on the same row as the IssueTriage panel name.

Workflow

1. Connect – Run Issue Triage: Connect Repository and complete the device-code sign-in.
2. Explore – Use the sidebar or matrix to find high-value candidates or risky outliers.
3. Assess – Trigger Issue Triage: Assess Selected Issue; IssueTriage gathers CLI context, calls the LLM, and updates history.
4. Act – Share results via optional GitHub comment, continue refinement, or launch automation if enabled.

Quick Start (VS Code & Cursor)

1. Install IssueTriage from the marketplace (works identically in Cursor).
2. Open the command palette → Issue Triage: Connect Repository → authorize in the browser.
3. Choose your LLM mode:
   • Remote (default) – Requests go through the hosted Cloudflare worker, no local key required.
   • Local – Provide your own OpenRouter key via .env (ISSUETRIAGE_OPENROUTER_API_KEY) or settings.
4. Open the panel (Issue Triage: Open Panel) or sidebar icon, select a repository, and run your first assessment.

Configuration Highlights

LLM Modes

// settings.json
{
  "issuetriage.assessment.llmMode": "remote",           // or "local"
  "issuetriage.assessment.remoteEndpoint": "",          // override worker URL (remote mode)
  "issuetriage.assessment.standardModel": "openai/gpt-5-mini",
  "issuetriage.assessment.premiumModel": "openai/gpt-5",
  "issuetriage.assessment.usePremiumModel": false
}

Custom CLI Context Tools

{
  "issuetriage.cliTools": [
    {
      "id": "test-coverage",
      "title": "Test Coverage Report",
      "command": "npm",
      "args": ["run", "test:coverage"],
      "cwd": "${workspaceRoot}",
      "autoRun": true,
      "refreshIntervalMs": 300000,
      "enabled": true
    }
  ]
}

Tokens supported: ${workspaceRoot}, ${workspaceFolder}, ${extensionRoot}.

Risk Intelligence Tuning

{
  "issuetriage.risk.lookbackDays": 180,
  "issuetriage.risk.labelFilters": ["bug", "feature"],
  "issuetriage.risk.publishComments": true
}

Automation Launch Guard

{
  "issuetriage.automation.launchEnabled": false
}

Only enable when your downstream AI coding agent integration is ready.

Command Palette Reference

Readiness Categories

Risk intelligence multiplies composite scores by 0.8 (high risk) or 0.9 (medium risk) when history indicates caution.

Telemetry & Privacy

• Local-first storage: assessment and risk databases live under VS Code global storage.
• GitHub comments: controlled via issuetriage.assessment.publishComments and issuetriage.risk.publishComments.
• Telemetry: disable entirely with issuetriage.telemetry.enabled = false.
• UsageTap integration: optional debug logs via issuetriage.telemetry.usagetapDebug.
• Source code never leaves your machine unless your custom CLI tools transmit data.

Troubleshooting Checklist

• Auth problems – Rerun Issue Triage: Sign Out, ensure the verification URL is reachable, and restart the editor.
• Missing API key (local mode) – Confirm .env or settings contain ISSUETRIAGE_OPENROUTER_API_KEY; reload after changes.
• Provider errors – Check OpenRouter status, switch to the standard model, or retry once rate limits reset.
• Empty issue list – Verify you have repo access and refresh via the command palette.
• Risk data missing – Ensure issues have linked PRs/commits within the configured lookback window.

Requirements

• VS Code 1.95.0+ or Cursor (latest builds).
• GitHub account with repository access.
• Network access to github.com and OpenRouter (or your configured proxy).
• Optional: OpenRouter API key for local LLM mode.

Release Notes (0.0.12)

• Opportunity Mix matrix webview inside the sidebar.
• Expanded assessment panel with risk intelligence, automation guard, and history timeline.
• ML Training tab with keyword backfill and dataset export workflows.
• CLI context tool orchestration with auto-run support and output history.
• Remote (worker-proxied) and local OpenRouter LLM modes with premium toggle.
• UsageTap LLM usage tracking with opt-in telemetry controls.

Triage faster, ship safer. Report issues or explore the code.