PromptSight AI

🧪 Beta — PromptSight is in internal beta. Features and thresholds may change; please report issues via @promptsight /feedback.

Prompt governance for VS Code — every token is checked before it is spent.

PromptSight sits between you and the LLM. Before a prompt is sent it tries to avoid the call entirely (local prompt cache), reduce it (lossless compression, deduplication, secret redaction), and govern it (token/cost thresholds, session budgets, team policy) — then stops so you decide. Nothing is sent until you choose.

User prompt (@promptsight …)  /  Agent task (#promptsight …)
        │
        ▼
① AVOID — Prompt Cache (local, error-signature ranked)
        ├─► error-signature & keyword match against solved prompts
        └─► confident hit? → 💾 reuse the answer — 0 tokens spent
        │
        ▼
② REDUCE — Optimizer (deterministic, zero-token)
        ├─► de-duplication & log-spam collapsing
        ├─► JSON → compact tabular re-encoding (30–60% smaller, lossless)
        └─► high-severity secret redaction (keys, tokens, JWTs)
        │
        ▼
③ GOVERN — Analyzer & Policy Gate
        ├─► token count + expected output   (heuristic, ≈¾ word/token)
        ├─► cost estimate                   (per-model pricing)
        ├─► ALLOW / WARN / BLOCK            (policy thresholds)
        ├─► session burn-down budget        (cumulative spend gate)
        └─► footprint estimate              (Wh / g CO₂e)
        │
        ▼
You decide:  💾 Use cached · ✅ Allow · ♻ Reframe · ⇄ Switch model · ⛔ Deny
        │
        ▼
Submit to LLM  (only if you choose) — the response is stored
back into the cache, so the next similar prompt is free.

Features

• 💾 Prompt cache (0-token reuse) — previously solved prompts live in a local JSON store, ranked by error signatures, acceptance rate, and recency (LRU-capped at 500 entries). A confident match is offered before any tokens are spent, and every model response is stored back automatically — the cache learns from your accept/reject clicks.
• 🧺 Compact data encoding — JSON payloads in your prompt are losslessly re-encoded in a compact tabular form (field list declared once, one row per item) in the optimized version. Deterministic encoding: the compression itself costs zero tokens.
• 🔐 Secret redaction on Reframe — high-severity secrets (API keys, bearer tokens, private keys, JWTs) are masked in the optimized prompt, so Reframe never leaks credentials.
• 🔥 Session burn-down — cumulative token spend is tracked per VS Code session; exceeding promptsight.sessionBudgetTokens requires explicit confirmation before every send (in chat and the agent gate).
• 🏛 Repo-committed policy — commit .promptsight/policy.json to your repo and the whole team inherits the same thresholds and allowed models via git.
• 🧠 Model routing — one click to act on it — complexity-based suggestions to downgrade (or upgrade) the model tier for the task. When a better-fitting model is suggested, a ⇄ Switch model button (or /switchmodel) sends the request via the suggested model — re-priced against it — without touching your chat settings. (VS Code does not let extensions change the chat window's model picker; PromptSight routes the request through the suggested model instead, which is what determines tokens and cost.)
• 📝 Built-in feedback loop — @promptsight /feedback up|down <comment> records ratings locally during rollout; /feedback summary shows satisfaction %, per-feature 👍/👎 counts, and recent comments to guide future development.
• 🌱 Footprint estimate — every report includes a rough energy/CO₂e estimate for the request.
• 🛡 Two surfaces — @promptsight chat participant (Ask mode) and #promptsight Allow/Deny/Reframe gate (Agent mode).

Everything runs locally and in-process — no Python, no subprocesses, no network calls, and nothing leaves your machine without your explicit decision.

Installation

Requirements: VS Code 1.95+ with GitHub Copilot Chat. Nothing else — no Python, no npm install; the analyzer runs in-process.

From the Marketplace (recommended):

1. Open the Extensions panel (Ctrl+Shift+X)
2. Search for PromptSight AI and click Install
3. Open Copilot Chat (Ctrl+Alt+I) and type @promptsight hello — the participant appears in the suggestions list

Or from the command line:

code --install-extension HemantSagar-MS.promptsight-preflight

Building from source (contributors):

git clone https://github.com/hemantsagar_microsoft/PromptSight_AI.git
cd PromptSight_AI
powershell -ExecutionPolicy Bypass -File build-vsix.ps1
code --install-extension .\promptsight-preflight-<version>.vsix --force

Then reload VS Code (Ctrl+Shift+P → Developer: Reload Window).

How to Use

In Copilot Chat — @promptsight

1. Open Copilot Chat (Ctrl+Alt+I)
2. Make sure you are in Ask mode (not Agent mode)
3. Type @promptsight followed by your prompt:

   @promptsight Summarize the entire history of the Roman Empire in detail
   

4. PromptSight shows a pre-flight report with token count, estimated cost, and status. Nothing is sent to the model yet.
5. Choose an action from the buttons shown under the report:

Slash command shortcuts:

After /cached, 👍/👎 rating buttons appear automatically — the cache learns which answers are worth reusing.

Model switching in practice: if you ask a simple question while a premium model (e.g. Claude Opus) is selected, the pre-flight report says "You can likely downgrade to a small / fast model such as 'gpt-4.1-mini' to cut cost" and shows the ⇄ Switch model button. Clicking it sends this request through the suggested model. The suggested model must be enabled in the chat model picker ("Manage Models…") — if it isn't, PromptSight tells you and sends nothing.

In Agent Mode — #promptsight

1. Switch Copilot Chat to Agent mode
2. Add #promptsight to your message:

   #promptsight Refactor all files in this repo to use async/await
   

3. A gate dialog appears with Allow, Reframe, and Deny options:
   • Allow — agent proceeds with your original request
   • Reframe — agent uses the optimized prompt instead
   • Deny — agent stops; adjust your prompt and retry

Settings

Open VS Code Settings (Ctrl+,) and search for promptsight:

Team policy via git

Create .promptsight/policy.json in your repository root and commit it — every teammate with PromptSight gets the same governance rules automatically:

{
  "warn_total_tokens": 8000,
  "block_total_tokens": 12000,
  "hard_block": true,
  "allowed_models": ["gpt-4.1", "gpt-4.1-mini", "gpt-4o-mini"],
  "max_duplicate_ratio": 0.2
}

Troubleshooting

Uninstall

code --uninstall-extension HemantSagar-MS.promptsight-preflight

Then reload VS Code: Ctrl+Shift+P → Developer: Reload Window.

Running the Tests

node --test test/

The suite (109 tests) covers the analyzer units, a 16-scenario validation corpus of technical and non-technical prompt types (code, logs, Terraform, SQL, JSON, transcripts, emails, marketing copy, PII/secrets…), model-selection impact (cost scaling, downgrade/upgrade routing), the feedback store, and a mocked-vscode smoke test of the chat participant (switch-model routing included).

Compare token/cost impact of different models across the whole corpus:

npm run eval:models          # or: node scripts/model-impact.js
node scripts/model-impact.js --models gpt-4.1-mini,claude-opus-4

The report shows per-scenario cost by model and where routing advice recommends a cheaper (cost) or stronger (output quality) model.

Repository Layout

PromptSight_AI/
├── package.json              Extension manifest (chat participant + agent tool)
├── extension.js              Chat participant, agent gate, cache UX, session budget, model switch, feedback
├── build-vsix.ps1            Packages the extension into a .vsix
├── scripts/
│   └── model-impact.js       Model-selection impact report over the validation corpus
├── src/                      In-process analyzer (no external dependencies)
│   ├── analyzer.js           Pre-flight orchestrator (tokens → cost → policy → optimization)
│   ├── tokenizer.js          Token counting + output estimation
│   ├── classifier.js         Input type / complexity / duplication detection
│   ├── costEstimator.js      Per-model pricing → cost
│   ├── policyValidator.js    ALLOW / WARN / BLOCK decision
│   ├── optimizer.js          Safe cleanup, model routing, secret detection/redaction
│   ├── promptCache.js        Local JSON prompt-solution cache (0-token reuse, LRU-capped)
│   ├── feedbackStore.js      Local rollout-feedback store (ratings + summary)
│   └── compactEncoder.js     JSON → compact tabular lossless compression
├── analyzer/
│   └── config/
│       ├── policy.json       Threshold and rule configuration
│       ├── pricing.json      Per-model token pricing
│       └── model_rules.json  Model routing rules
└── test/
    ├── analyzer.test.js      Analyzer unit tests
    ├── validation.test.js    Recommendation validation over the prompt corpus
    ├── modelImpact.test.js   Model-selection impact on tokens/cost/routing
    ├── feedbackStore.test.js Feedback store tests
    ├── extensionSmoke.test.js Chat participant smoke test (mocked vscode)
    └── fixtures/
        └── promptCorpus.js   16 technical + non-technical validation scenarios

Notes

• Token counts are heuristic (≈¾ word per token); output-token, cost, and CO₂e figures are estimates for pre-flight awareness only.
• The analyzer runs fully in-process — no Python, no subprocesses, no network calls.
• The optimizer only performs deterministic, zero-token edits: deduplication, whitespace cleanup, repeated-line collapsing, lossless compact re-encoding of JSON, and secret masking. No LLM is used to optimize prompts.
• Secret detection is best-effort; high-severity matches are masked in the optimized prompt and always redacted in the report.
• The prompt cache is a local JSON file (per-machine, under VS Code global storage, capped at 500 entries). Nothing is uploaded anywhere.
• Rollout feedback (/feedback) is also a local JSON file (capped at 1000 entries) — nothing is uploaded anywhere.
• Model switching routes the request through the suggested model via the VS Code language-model API; the chat window's model picker UI itself cannot be changed by extensions and keeps showing your manual selection.
• GitHub Copilot Chat is required to use the @promptsight participant and #promptsight tool.