PromptSight AI

Prompt governance for VS Code — every token is checked before it is spent.

PromptSight sits between you and the LLM. Before a prompt is sent it tries to avoid the call entirely (local prompt cache), reduce it (lossless compression, deduplication, secret redaction), and govern it (token/cost thresholds, session budgets, team policy) — then stops so you decide. Nothing is sent until you choose.

User prompt (@promptsight …)  /  Agent task (#promptsight …)
        │
        ▼
① AVOID — Prompt Cache (local, error-signature ranked)
        ├─► error-signature & keyword match against solved prompts
        └─► confident hit? → 💾 reuse the answer — 0 tokens spent
        │
        ▼
② REDUCE — Optimizer (deterministic, zero-token)
        ├─► de-duplication & log-spam collapsing
        ├─► JSON → compact tabular re-encoding (30–60% smaller, lossless)
        └─► high-severity secret redaction (keys, tokens, JWTs)
        │
        ▼
③ GOVERN — Analyzer & Policy Gate
        ├─► token count + expected output   (heuristic, ≈¾ word/token)
        ├─► cost estimate                   (per-model pricing)
        ├─► ALLOW / WARN / BLOCK            (policy thresholds)
        ├─► session burn-down budget        (cumulative spend gate)
        └─► footprint estimate              (Wh / g CO₂e)
        │
        ▼
You decide:  💾 Use cached · ✅ Allow · ♻ Reframe · ⛔ Deny
        │
        ▼
Submit to LLM  (only if you choose) — the response is stored
back into the cache, so the next similar prompt is free.

Features

• 💾 Prompt cache (0-token reuse) — previously solved prompts live in a local JSON store, ranked by error signatures, acceptance rate, and recency (LRU-capped at 500 entries). A confident match is offered before any tokens are spent, and every model response is stored back automatically — the cache learns from your accept/reject clicks.
• 🧺 Compact data encoding — JSON payloads in your prompt are losslessly re-encoded in a compact tabular form (field list declared once, one row per item) in the optimized version. Deterministic encoding: the compression itself costs zero tokens.
• 🔐 Secret redaction on Reframe — high-severity secrets (API keys, bearer tokens, private keys, JWTs) are masked in the optimized prompt, so Reframe never leaks credentials.
• 🔥 Session burn-down — cumulative token spend is tracked per VS Code session; exceeding promptsight.sessionBudgetTokens requires explicit confirmation before every send (in chat and the agent gate).
• 🏛 Repo-committed policy — commit .promptsight/policy.json to your repo and the whole team inherits the same thresholds and allowed models via git.
• 🧠 Model routing — complexity-based suggestions to downgrade (or upgrade) the model tier for the task.
• 🌱 Footprint estimate — every report includes a rough energy/CO₂e estimate for the request.
• 🛡 Two surfaces — @promptsight chat participant (Ask mode) and #promptsight Allow/Deny/Reframe gate (Agent mode).

Everything runs locally and in-process — no Python, no subprocesses, no network calls, and nothing leaves your machine without your explicit decision.

Installation

Requirements: VS Code 1.95+ with GitHub Copilot Chat. Nothing else — no Python, no npm install; the analyzer runs in-process.

From the Marketplace (recommended):

1. Open the Extensions panel (Ctrl+Shift+X)
2. Search for PromptSight AI and click Install
3. Open Copilot Chat (Ctrl+Alt+I) and type @promptsight hello — the participant appears in the suggestions list

Or from the command line:

code --install-extension HemantSagar.promptsight-ai

Building from source (contributors):

git clone https://github.com/hemantsagar_microsoft/PromptSight_AI.git
cd PromptSight_AI
powershell -ExecutionPolicy Bypass -File build-vsix.ps1
code --install-extension .\promptsight-ai-<version>.vsix --force

Then reload VS Code (Ctrl+Shift+P → Developer: Reload Window).

How to Use

In Copilot Chat — @promptsight

1. Open Copilot Chat (Ctrl+Alt+I)
2. Make sure you are in Ask mode (not Agent mode)
3. Type @promptsight followed by your prompt:

   @promptsight Summarize the entire history of the Roman Empire in detail
   

4. PromptSight shows a pre-flight report with token count, estimated cost, and status. Nothing is sent to the model yet.
5. Choose an action from the buttons shown under the report:

Slash command shortcuts:

After /cached, 👍/👎 rating buttons appear automatically — the cache learns which answers are worth reusing.

In Agent Mode — #promptsight

1. Switch Copilot Chat to Agent mode
2. Add #promptsight to your message:

   #promptsight Refactor all files in this repo to use async/await
   

3. A gate dialog appears with Allow, Reframe, and Deny options:
   • Allow — agent proceeds with your original request
   • Reframe — agent uses the optimized prompt instead
   • Deny — agent stops; adjust your prompt and retry

Settings

Open VS Code Settings (Ctrl+,) and search for promptsight:

Team policy via git

Create .promptsight/policy.json in your repository root and commit it — every teammate with PromptSight gets the same governance rules automatically:

{
  "warn_total_tokens": 8000,
  "block_total_tokens": 12000,
  "hard_block": true,
  "allowed_models": ["gpt-4.1", "gpt-4.1-mini", "gpt-4o-mini"],
  "max_duplicate_ratio": 0.2
}

Troubleshooting

Uninstall

code --uninstall-extension HemantSagar.promptsight-ai

Then reload VS Code: Ctrl+Shift+P → Developer: Reload Window.

Running the Tests

node --test test/

Repository Layout

PromptSight_AI/
├── package.json              Extension manifest (chat participant + agent tool)
├── extension.js              Chat participant, agent gate, cache UX, session budget
├── build-vsix.ps1            Packages the extension into a .vsix
├── src/                      In-process analyzer (no external dependencies)
│   ├── analyzer.js           Pre-flight orchestrator (tokens → cost → policy → optimization)
│   ├── tokenizer.js          Token counting + output estimation
│   ├── classifier.js         Input type / complexity / duplication detection
│   ├── costEstimator.js      Per-model pricing → cost
│   ├── policyValidator.js    ALLOW / WARN / BLOCK decision
│   ├── optimizer.js          Safe cleanup, model routing, secret detection/redaction
│   ├── promptCache.js        Local JSON prompt-solution cache (0-token reuse, LRU-capped)
│   └── compactEncoder.js     JSON → compact tabular lossless compression
├── analyzer/
│   └── config/
│       ├── policy.json       Threshold and rule configuration
│       ├── pricing.json      Per-model token pricing
│       └── model_rules.json  Model routing rules
└── test/
    └── analyzer.test.js      node:test suite

Notes

• Token counts are heuristic (≈¾ word per token); output-token, cost, and CO₂e figures are estimates for pre-flight awareness only.
• The analyzer runs fully in-process — no Python, no subprocesses, no network calls.
• The optimizer only performs deterministic, zero-token edits: deduplication, whitespace cleanup, repeated-line collapsing, lossless compact re-encoding of JSON, and secret masking. No LLM is used to optimize prompts.
• Secret detection is best-effort; high-severity matches are masked in the optimized prompt and always redacted in the report.
• The prompt cache is a local JSON file (per-machine, under VS Code global storage, capped at 500 entries). Nothing is uploaded anywhere.
• GitHub Copilot Chat is required to use the @promptsight participant and #promptsight tool.