LLM Watch

Shift-left tooling for LLM app development in VS Code.

LLM Watch helps you catch LLM observability and safety issues while writing code, not after incidents in production.

Current MVP Features

• Detects baseline LLM SDK call sites in TypeScript and JavaScript:
  • openai.chat.completions.create(...)
  • anthropic.messages.create(...)
• Adds diagnostics for uninstrumented LLM calls.
• Provides CodeLens and Quick Fix to wrap calls in OpenTelemetry GenAI span templates.
• Includes a command to generate a local OpenTelemetry ConsoleSpanExporter bootstrap file.
• Adds guardrails diagnostics for:
  • Unsafe prompt concatenation with likely user input.
  • PII-like values in prompt-like fields (email, phone, card-like patterns).
• Adds prompt management basics:
  • Prompt file discovery (.prompt.yaml, .prompt.md)
  • Prompt sidebar tree view
  • Prompt variable preview command

Commands

• LLM Watch: Wrap LLM call in OTel span
• LLM Watch: Set OpenAI API Key
• LLM Watch: Create OTel Console Bootstrap File
• LLM Watch: Preview Prompt Variables

Settings

• llm-watch.enabled
• llm-watch.observability.enabled
• llm-watch.guardrails.enabled
• llm-watch.gpu.enabled
• llm-watch.prompts.enabled
• llm-watch.telemetry.enabled

Privacy & Telemetry

LLM Watch includes opt-in anonymous telemetry to help improve the extension. You are asked once on first activation; you can change the choice any time in Settings.

What is collected (when opted in)

• Extension version, OS type (e.g. darwin)
• Anonymous event counts: CodeLens shown, span wrapped, guardrail rule fired, error type
• No user identifiers, no workspace paths

What is never collected

• Your code, prompts, or LLM outputs
• Model names, token counts, or API costs
• API keys or any secrets
• Personally identifiable information

Disabling telemetry

Set llm-watch.telemetry.enabled to false in VS Code Settings, or choose No Thanks on first activation.

Local Development

1. Install dependencies:

npm install

2. Build and test:

npm run release:check

3. Run in Extension Development Host:

• Use Run LLM Watch Extension or Run LLM Watch (Isolated) from Run and Debug.

Smoke Test Files

Use these files in the repo to validate behavior quickly:

• smoke/openai-unsafe.ts
• smoke/anthropic-pii.ts
• smoke/template.prompt.md

Packaging

Create a VSIX package:

npm run release:package

This produces a .vsix file you can install locally via:

• VS Code Command Palette -> Extensions: Install from VSIX...

Status

This is an MVP release focused on shift-left observability + guardrails for LLM call sites. Additional roadmap features will be added in future releases.