PromptSight AI

Prompt governance for VS Code — every token is checked before it is spent.

PromptSight sits between you and the LLM. Before a prompt is sent it tries to avoid the call entirely (local prompt cache), reduce it (lossless compression, exact-repeat summaries, secret redaction), and govern it (token/cost thresholds, session budgets, team policy, personal-data masking) — then stops so you decide. Nothing is sent until you choose.

User prompt (@promptsight …)  /  Agent task (#promptsight …)
        │
        ▼
① AVOID — Prompt Cache (local, error-signature ranked)
        ├─► error-signature & keyword match against solved prompts
        └─► confident hit? → 💾 reuse the answer — 0 tokens spent
        │
        ▼
② REDUCE — Optimizer (deterministic, zero-token)
        ├─► exact log/transcript repeats summarized with counts
        ├─► JSON → compact tabular re-encoding (30–60% smaller, lossless)
        ├─► high-severity secret redaction (keys, tokens, JWTs)
        └─► personal data → placeholders (names, SSN, email… → [NAME_1])
        │
        ▼
③ GOVERN — Analyzer & Policy Gate
        ├─► token count + expected output   (heuristic, ≈¾ word/token)
        ├─► cost estimate                   (per-model pricing)
        ├─► ALLOW / WARN / BLOCK            (policy thresholds)
        ├─► session burn-down budget        (charged with actual usage)
        ├─► reply cap                       (worst case = prompt + cap)
        └─► footprint estimate              (Wh / g CO₂e)
        │
        ▼
You decide:  💾 Use cached · ✅ Allow · ♻ Reframe · ⇄ Switch model · ⛔ Deny
        │
        ▼
Submit to LLM  (only if you choose) — the response is stored
back into the cache, so the next similar prompt is free.

Features

• 📎 Attachment-aware pre-flight — files or selections attached via VS Code's native 📎 picker are analyzed just like typed text and included in the overall token and cost estimates; the report's 📎 Attachment row shows Yes or No (the open editor VS Code adds on its own doesn't count, and binary files such as images are skipped)
• 💾 Prompt cache (0-token reuse), enabled by default — prompts you've already solved are kept in a local JSON store, ranked by error signatures, acceptance rate, and recency (LRU-capped at 500 entries). A confident match is offered before any tokens are spent, and every model response (with high-severity secrets redacted) is stored back automatically — the cache learns from your accept/reject clicks. Entries recorded against an attached file are pinned to that file's content hash, so a same-named file in a different project can never return the wrong project's cached answer.
• 🧺 Compact data encoding — JSON payloads in your prompt are losslessly re-encoded in a compact tabular form (field list declared once, one row per item) in the optimized version. Deterministic encoding: the compression itself costs zero tokens.
• 🔐 Secret redaction on Reframe — high-severity secrets (API keys, bearer tokens, private keys, JWTs) are masked in the optimized prompt, so Reframe never leaks credentials.
• 🛡️ Personal data masked on every send — names, SSNs, Aadhaar and PAN numbers, passport and driver's licence numbers, card, IBAN and account numbers, email addresses, phone numbers, street addresses, dates of birth, and public IP addresses are swapped for placeholders like [NAME_1] before anything reaches the model; the reply you see has the real values put back. Detection is local and deterministic (formats, checksums, labelled fields). Names come from context (titles, greetings, sign-offs, Name: fields), a built-in list of common first names, and promptsight.piiNames for names the rules can't recognise. The pre-flight report's 🛡️ Personal data row shows what will be masked, and personal data is never written to the local cache.
• 🔥 Session burn-down — the tokens each send actually used (prompt + reply) are tracked per VS Code session; a send that could go over promptsight.sessionBudgetTokens requires explicit confirmation first (in chat and the agent gate — agent-mode sends are charged at their estimate, since the agent's own replies can't be observed).
• 🧢 Reply cap + actual usage — a few words can produce a very long answer, so no estimate can know a reply's length in advance. Every reply sent through @promptsight stops at promptsight.maxOutputTokens (8,000 by default), so the most a send can use is known before it goes out, and a ▶ Continue button fetches the rest. Each reply ends with its estimated vs actual token usage.
• 🏛 Repo-committed policy — commit .promptsight/policy.json to your repo and the whole team inherits the same thresholds and allowed models via git.
• 🧠 Model routing — one click to act on it — complexity-based suggestions to downgrade (or upgrade) the model tier for the task. When a better-fitting model is suggested, a ⇄ Switch model button (or /switchmodel) sends the request via the suggested model — re-priced against it — without touching your chat settings. (VS Code does not let extensions change the chat window's model picker; PromptSight routes the request through the suggested model instead, which is what determines tokens and cost.) For high-complexity tasks the suggestion is HydraFusion (GitHub's multi-model orchestration); if the chat doesn't offer it to extensions, Switch model falls back to gpt-5.6-sol and says so. HydraFusion has no published rate, so its cost is shown at your promptsight.defaultModel rates.
• 🔁 Works with Copilot's Auto — VS Code doesn't let extensions send through Auto, so PromptSight sends via your promptsight.defaultModel (or the first available model) and tells you which; cost is shown at that model's rates, labelled as such.
• 🛟 Clear failure signaling — if the model returns no text or errors mid-request, PromptSight reports it explicitly instead of silently treating it as a completed send.
• 📝 Built-in feedback — @promptsight /feedback up|down <comment> records your ratings of PromptSight on your machine; /feedback summary shows satisfaction %, per-feature 👍/👎 counts, and recent comments.
• 🌱 Footprint estimate — every report includes a rough energy/CO₂e estimate for the request.
• 📊 Usage dashboard — @promptsight /dashboard (or the PromptSight: Open Dashboard command) groups results into clearly labeled Token savings and Cost savings sections (Optimization / Cache reuse / Total), plus cache hit rate and model-switch adoption, computed from your own local usage history. Cache-reuse potential (hits shown at pre-flight but not used) is still tracked in the background but is not shown in the UI for now.

Everything runs locally and in-process — no Python, no subprocesses, no network calls, and nothing leaves your machine without your explicit decision.

Installation

Requirements: VS Code 1.95+ with GitHub Copilot Chat. Nothing else — no Python, no npm install; the analyzer runs in-process.

From the Marketplace (recommended):

1. Open the Extensions panel (Ctrl+Shift+X)
2. Search for PromptSight AI and click Install
3. Open Copilot Chat (Ctrl+Alt+I) and type @promptsight hello — the participant appears in the suggestions list

Or from the command line:

code --install-extension HemantSagar-MS.promptsight-preflight

Building from source (contributors):

git clone https://github.com/hemantsagar_microsoft/PromptSight_AI.git
cd PromptSight_AI
powershell -ExecutionPolicy Bypass -File build-vsix.ps1
code --install-extension .\promptsight-preflight-<version>.vsix --force

Then reload VS Code (Ctrl+Shift+P → Developer: Reload Window).

How to Use

In Copilot Chat — @promptsight

1. Open Copilot Chat (Ctrl+Alt+I)
2. Make sure you are in Ask mode (not Agent mode)
3. Type @promptsight followed by your prompt:

   @promptsight Summarize the entire history of the Roman Empire in detail
   

4. PromptSight shows a pre-flight report with token count, estimated cost, status, and whether anything is attached (📎 Attachment: Yes/No). Nothing is sent to the model yet.
5. Choose an action from the buttons shown under the report:

Allow and Reframe remain available even when there are no automatic savings; Reframe keeps the send optimized label and shows token savings only when positive, sending unchanged text without claiming savings if the optimizer found nothing to change. Switch model and Reframe + Switch model always appear too — if the current model already fits the task, clicking either just replies that no switch is needed rather than opening a manual model picker. Use cached solution appears only when a confident cache match exists.

When you send, the first line splits the estimate into Input tokens and Output tokens; after Reframe the input shows the condensed count, for example:

PromptSight: ALLOW · Input tokens: ~160 (condensed from ~560) and Output tokens: ~480 (reply capped at 8,000) · saved ~600 tokens · sending optimized prompt…

The reply then ends with the actual usage, split the same way: estimated ~640 → actual 1,210 tokens (input 160 + output 1,050).

Slash command shortcuts:

After /cached, 👍/👎 rating buttons appear automatically — the cache learns which answers are worth reusing.

Model switching in practice: if you ask a simple question while a premium model (e.g. Claude Opus) is selected, the pre-flight report says "You can likely downgrade to a small / fast model such as 'gpt-4.1-mini' to cut cost" and shows the ⇄ Switch model button. Clicking it sends this request through the suggested model. The suggested model must be enabled in the chat model picker ("Manage Models…") — if it isn't, PromptSight tells you and sends nothing.

In Agent Mode — #promptsight

1. Switch Copilot Chat to Agent mode
2. Add #promptsight to your message:

   #promptsight Refactor all files in this repo to use async/await
   

3. A gate dialog appears with Allow, Reframe, and Deny options:
   • Allow — agent proceeds with your original request
   • Reframe — agent uses the optimized prompt instead
   • Deny — agent stops; adjust your prompt and retry

The reply cap and the estimated-vs-actual usage apply to @promptsight only: in agent mode the agent's own replies don't pass through PromptSight, so they can't be capped or measured. Likewise the agent already has your message, so personal-data masking only protects the Reframe text PromptSight hands back to it.

After updating to VS Code 1.141

VS Code 1.141 can open new chats as Copilot (agent host) sessions, which run on models that only the agent host can use (ids such as agent-host-copilotcli:auto). Earlier releases quietly ran a @promptsight request in such a chat on the default Copilot model instead. Since 1.141 (microsoft/vscode#339307), VS Code rejects the request before it reaches PromptSight:

The selected model 'agent-host-copilotcli:auto' is not available. Select a different model and try again.

Picking another model in that chat doesn't help, and no extension can intercept this error. PromptSight therefore fixes it on its own, so nobody has to click anything:

• When VS Code starts, PromptSight turns off chat.defaultToCopilotHarness and chat.editor.preferCopilotHarness in your user settings if either is on, and shows a short notice with an Undo button. New chats then open as regular Copilot chats, where you can type @promptsight.
• This happens once per install, and only when VS Code's own default turned these settings on. If you turned either on yourself, chose Don't Show Again on the notification in version 1.1.4, turn one back on later, or click Undo, PromptSight leaves your choice alone.
• Chats that are already open keep their type, so use a new chat afterwards. The setting applies to every new chat, not only PromptSight's.
• To apply the fix again later, run PromptSight: Fix @promptsight for New Chats from the Command Palette. It also opens a new chat with @promptsight typed in.

#promptsight isn't affected: tools don't depend on the chat's model, so the Allow / Deny / Reframe gate keeps working in Copilot (agent host) chats.

Settings

Open VS Code Settings (Ctrl+,) and search for promptsight:

Team policy via git

Create .promptsight/policy.json in your repository root and commit it — every teammate with PromptSight gets the same governance rules automatically (a teammate who sets promptsight.warnThreshold or promptsight.blockThreshold themselves keeps that threshold):

{
  "warn_total_tokens": 8000,
  "block_total_tokens": 12000,
  "hard_block": true,
  "allowed_models": ["gpt-4.1", "gpt-4.1-mini", "gpt-4o-mini"],
  "max_duplicate_ratio": 0.2
}

Troubleshooting

Uninstall

code --uninstall-extension HemantSagar-MS.promptsight-preflight

Then reload VS Code: Ctrl+Shift+P → Developer: Reload Window.

Running the Tests

npm test

The suite covers the analyzer units, personal-data masking, a 16-scenario validation corpus of technical and non-technical prompt types (code, logs, Terraform, SQL, JSON, transcripts, emails, marketing copy, PII/secrets…), model-selection impact (cost scaling, downgrade/upgrade routing), the feedback store, a mocked-vscode smoke test of the chat participant (switch-model routing, the reply cap, and ▶ Continue included), and the automatic fix for @promptsight in Copilot (agent host) chats.

Compare token/cost impact of different models across the whole corpus:

npm run eval:models          # or: node scripts/model-impact.js
node scripts/model-impact.js --models gpt-4.1-mini,claude-opus-4

The report shows per-scenario cost by model and where routing advice recommends a cheaper (cost) or stronger (output quality) model.

Repository Layout

PromptSight_AI/
├── package.json              Extension manifest (chat participant + agent tool)
├── extension.js              Chat participant, agent gate, cache UX, session budget, model switch, feedback
├── lib/                      VS Code-facing helpers used by extension.js
│   ├── config.js             Settings and repo-committed policy lookup
│   ├── attachments.js        Chat attachments (files, selections, inline text) → prompt text
│   ├── dashboard.js          /dashboard, /clearcache, and the dashboard webview
│   ├── feedback.js           /feedback ratings and summary
│   ├── format.js             Shared text, Markdown, and HTML formatting
│   └── chatDefault.js        One-time automatic fix for @promptsight in Copilot (agent host) chats
├── build-vsix.ps1            Packages the extension into a .vsix
├── scripts/
│   └── model-impact.js       Model-selection impact report over the validation corpus
├── src/                      In-process analyzer (one small bundled dependency, no network calls)
│   ├── analyzer.js           Pre-flight orchestrator (tokens → cost → policy → optimization)
│   ├── tokenizer.js          Token counting (real BPE for OpenAI models) + calibrated output estimation
│   ├── classifier.js         Input type / complexity / duplication detection
│   ├── costEstimator.js      Per-model pricing → cost
│   ├── policyValidator.js    ALLOW / WARN / BLOCK decision
│   ├── optimizer.js          Safe cleanup, model routing, secret detection/redaction
│   ├── piiMasker.js          Personal-data detection and reversible masking
│   ├── promptCache.js        Local JSON prompt-solution cache (0-token reuse, LRU-capped)
│   ├── feedbackStore.js      Local feedback store (ratings + summary)
│   ├── metricsStore.js       Local usage/telemetry store (savings, cache reuse, dashboard)
│   └── compactEncoder.js     JSON → compact tabular lossless compression
├── analyzer/
│   └── config/
│       ├── policy.json               Threshold and rule configuration
│       ├── pricing.json              Per-model token pricing
│       ├── model_rules.json          Model routing rules
│       └── tokenizer_encodings.json  Model name → BPE encoding table
└── test/
    ├── analyzer.test.js      Analyzer unit tests
    ├── validation.test.js    Recommendation validation over the prompt corpus
    ├── modelImpact.test.js   Model-selection impact on tokens/cost/routing
    ├── feedbackStore.test.js Feedback store tests
    ├── metricsStore.test.js  Usage-metrics store tests
    ├── promptCache.test.js   Prompt cache tests
    ├── piiMasker.test.js     Personal-data detection and masking tests
    ├── tokenizer.test.js     BPE/heuristic token counting + output calibration tests
    ├── extensionSmoke.test.js Chat participant smoke test (mocked vscode)
    ├── chatDefault.test.js   Automatic chat fix tests
    └── fixtures/
        └── promptCorpus.js   16 technical + non-technical validation scenarios

Notes

• Token counts use real BPE tokenization (via the bundled js-tiktoken) for OpenAI-family models — the exact count they're billed on. Every other vendor (Claude, Gemini, Grok, MAI) falls back to a heuristic estimate (≈¾ word per token). Very long unbroken runs of text (CJK paragraphs, separator lines, minified blobs) are counted from a sample. Output-token, cost, and CO₂e figures are estimates, blended with your own local usage history once enough samples accumulate, for pre-flight awareness only. Cost uses the rates of the model picked in the chat; when it has none (e.g. Copilot's Auto), the promptsight.defaultModel rates are used and the report says so.
• Input tokens are counted on exactly what @promptsight sends — your prompt plus attachments, with personal data masked — using the tokenizer of the model that answers (for Copilot's Auto, the model PromptSight sends through). So the pre-flight prompt size, the send line, and the usage line after the reply show the same input count; only the output was estimated. The actual usage is PromptSight's count of the text sent and the reply received; a provider may add a few tokens of its own message framing.
• The analyzer runs fully in-process — no Python, no subprocesses, no network calls at runtime. The one bundled dependency (js-tiktoken, a pure-JS BPE tokenizer with pre-computed rank-table data files) never phones home; it only ever reads from local node_modules.
• The optimizer only performs deterministic, zero-token edits: exact-repeat summaries, lossless compact re-encoding of JSON, and secret masking. No LLM is used to optimize prompts.
• Repetition summaries are limited to exact adjacent entries in clearly recognized logs or timestamped transcripts, with repetition counts retained. Different values and connections are never treated as duplicates. Source code, SQL, Terraform, configuration, mixed or uncertain content, and repetitions in separate contexts are left unchanged by this step. Preserving meaning takes priority over token savings.
• Secret detection is best-effort; high-severity matches are masked in the optimized prompt and always redacted in the report.
• Personal-data detection is local and best-effort. Formats, checksums (Luhn for cards, Verhoeff for Aadhaar, mod-97 for IBAN) and labelled fields are reliable; a name that matches none of the rules and isn't in promptsight.piiNames can still get through. Private and reserved IP addresses and CIDR blocks identify no one, so they are left as they are.
• The prompt cache is enabled by default and is a local JSON file (per-machine, under VS Code global storage, capped at 500 entries). It stores only the derived keyword list (not the raw prompt/file text), a content hash per attached file (used to stop cross-project matches on same-named files, never the file content itself), and the model's response with personal data masked and high-severity secrets redacted. Nothing is uploaded anywhere; disable promptsight.cacheEnabled if you work with sensitive content.
• Feedback (/feedback) is also a local JSON file (capped at 1000 entries) — nothing is uploaded anywhere.
• Model switching routes the request through the suggested model via the VS Code language-model API; the chat window's model picker UI itself cannot be changed by extensions and keeps showing your manual selection.
• GitHub Copilot Chat is required to use the @promptsight participant and #promptsight tool.