WATCHTOWER

AI hallucination prevention for VS Code. Watchtower sits between you and AI models, catching uncertainty in your prompts before they cause hallucinations, and scanning AI responses after for fabrication, speculation, and unverified claims.

How It Works

Input Side — Prompt Analysis

When you ask @watchtower a question in VS Code Chat, it analyzes your prompt for hallucination risk signals before forwarding to the AI.

Detection categories:

If uncertainty score >= 4, Watchtower asks clarifying questions before forwarding your prompt. Your answers get bundled into an enriched prompt that produces far more accurate responses.

If the prompt is clean, it passes straight through — zero friction.

Output Side — Response Scanner (v0.3.0+)

After the AI responds, Watchtower scans the full response for hallucination signals and appends a severity score.

Scoring: 1-5 scale

What it catches:

• Fabricated metrics — naming APIs, libraries, or data that don't exist
• Fabricated causation — "the crash happened because users migrated" stated as fact without evidence
• Opinions — "I think", "in my opinion" (AI has no opinions)
• Value judgments — "great idea", "solid project", "excellent choice"
• Speculation — "probably", "likely", "seems like"
• Predictions — "this will scale well", "expect performance gains"
• Recommendations — "I'd recommend", "you should use" without presenting alternatives
• False authority — "historically", "research shows" without a citation
• False precision — "73% probability" pulled from nowhere
• Unverified claims — "data shows", "data confirms" without proof
• Subjective quality — "elegant", "practical", "fun to build"

Usage

VS Code Chat

Type @watchtower in VS Code Chat followed by your question:

@watchtower What's the latest way to configure ESLint with TypeScript?

Watchtower will either:

1. Ask clarifying questions if uncertainty is detected, then enrich your prompt
2. Pass through if the prompt is clean

After the AI responds, a score banner appears if any hallucination signals are found.

Commands

Claude Code Hook

Watchtower also runs as a Claude Code hook for output scanning. See Claude Code Integration below.

Claude Code Integration

Watchtower includes a PostToolUse hook that scans Claude's outputs in real-time. When Claude writes code, runs scripts, or generates text, the hook checks for hallucination signals and injects corrective context.

Hook file: ~/.claude/hooks/watchtower-output.js

Setup in ~/.claude/settings.json:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": ".*",
        "hooks": [
          {
            "type": "command",
            "command": "node \"~/.claude/hooks/watchtower-output.js\"",
            "timeout": 10
          }
        ]
      }
    ]
  }
}

The hook uses the same 1-5 scoring system:

• Score 1-2: Logged or soft reminder injected
• Score 3: Claude must display the alert
• Score 4: Claude must display + correct flagged items
• Score 5: Claude must STOP and retract flagged items

Architecture

User Prompt
    |
    v
[Input Detector] ── analyzes prompt for uncertainty signals
    |
    |── Clean? ──> Forward to AI ──> [Output Scanner] ──> Show response + score
    |
    |── Uncertain? ──> Ask clarifying questions
                            |
                            v
                     User answers
                            |
                            v
                    [Enricher] builds enhanced prompt
                            |
                            v
                     Forward to AI ──> [Output Scanner] ──> Show response + score

Requirements

• VS Code 1.94+
• GitHub Copilot extension (for the LLM backend)

Installation

From Marketplace: Search "WATCHTOWER" in VS Code Extensions, or visit the Marketplace listing.

From VSIX:

code --install-extension BLB-watchtower-0.4.0.vsix