Azure OpenAI PromptOps Manager
Version prompts in Git, run side-by-side across GPT-4 / GPT-4o / GPT-4o-mini, compare token cost + latency + output quality, and detect regressions — all inside VS Code.
Every team using Azure OpenAI manages prompts manually — in Notepad, Slack threads, or random JSON files. There is no versioning, no A/B comparison, no cost-per-prompt tracking, and no regression testing. This extension fixes all of that.
Features
Prompt Library & Versioning
- Create, edit, and organize prompts in a dedicated
.promptops/ directory inside your workspace
- Version every change with change notes — full Git-friendly history
- Diff any two versions side by side to see exactly what changed
- Template variables using
{{variable}} syntax with default values
- Tag-based organization for filtering and grouping prompts
- Import / Export your entire prompt library as JSON
Side-by-Side Model Comparison
- Run any prompt across GPT-4, GPT-4o, and GPT-4o-mini simultaneously
- Rich comparison panel showing output, cost, latency, token usage, and ms/token for each model
- Visual cost and latency bars for instant readability
- Best-value highlighting to identify the most cost-efficient model
Token Cost & Latency Tracking
- Real-time cost calculation per execution using configurable per-model rates
- Cost Dashboard with 24h / 7-day / all-time summaries
- Cost breakdown by model and by prompt with visual distribution bars
- Status bar indicator showing running cost total and execution count
Regression Testing
- Create baselines from any prompt version on any model
- Run regressions against new versions to detect output quality changes
- Jaccard bigram similarity scoring with configurable tolerance thresholds
- Pass/fail reporting with full baseline vs. test output comparison
Getting Started
1. Install the Extension
Install from the VS Code Marketplace or from the .vsix file.
Run PromptOps: Configure Azure OpenAI Endpoint from the Command Palette (Ctrl+Shift+P) and enter:
- Your Azure OpenAI endpoint URL (e.g.
https://myresource.openai.azure.com)
- Your API key
- Deployment names for each model (GPT-4, GPT-4o, GPT-4o-mini)
3. Create Your First Prompt
- Click the PromptOps icon in the Activity Bar
- Run
PromptOps: Create New Prompt
- Enter a name, description, system prompt, and user prompt template
- Use
{{variable}} syntax for dynamic content
4. Run & Compare
- Single model:
PromptOps: Run Prompt
- Multi-model comparison:
PromptOps: Compare Across Models
- View costs:
PromptOps: View Cost Dashboard
- Regression test:
PromptOps: Run Regression Test
Commands
| Command |
Description |
PromptOps: Create New Prompt |
Create a new prompt with system/user templates and variables |
PromptOps: Edit Prompt |
Open a prompt's JSON file for editing |
PromptOps: Run Prompt |
Execute a prompt on a single model |
PromptOps: Compare Across Models |
Run a prompt on multiple models simultaneously |
PromptOps: Run Regression Test |
Test a prompt version against baselines |
PromptOps: View Cost Dashboard |
Open the cost analytics dashboard |
PromptOps: Save Prompt Version |
Save the current prompt state as a new version |
PromptOps: Diff Prompt Versions |
Compare two versions of a prompt |
PromptOps: Export Prompt Library |
Export all prompts and history to JSON |
PromptOps: Import Prompts |
Import prompts from a JSON file |
PromptOps: Configure Azure OpenAI Endpoint |
Set up your endpoint, API key, and deployments |
PromptOps: Refresh Prompt Library |
Refresh the sidebar tree views |
Configuration
| Setting |
Default |
Description |
promptOps.endpoint |
"" |
Azure OpenAI endpoint URL |
promptOps.apiKey |
"" |
Azure OpenAI API key |
promptOps.deployments |
{} |
Model-to-deployment-name mapping |
promptOps.apiVersion |
2024-06-01 |
Azure OpenAI API version |
promptOps.promptsDir |
.promptops |
Directory for prompt storage |
promptOps.defaultModel |
gpt-4o |
Default model for single runs |
promptOps.maxTokens |
2048 |
Default max tokens per completion |
promptOps.temperature |
0.7 |
Default temperature |
promptOps.costPerMillionInput |
GPT-4: $30, GPT-4o: $2.50, GPT-4o-mini: $0.15 |
Input token cost rates |
promptOps.costPerMillionOutput |
GPT-4: $60, GPT-4o: $10, GPT-4o-mini: $0.60 |
Output token cost rates |
Prompt Storage Structure
Prompts are stored in your workspace under .promptops/:
.promptops/
├── prompts/ # Prompt definition JSON files
│ ├── code-review-abc1.json
│ └── summarizer-def2.json
├── history/ # Version history per prompt
│ ├── code-review-abc1.json
│ └── summarizer-def2.json
├── baselines/ # Regression test baselines
│ └── code-review-abc1.json
└── cost-log.json # Running cost log
All files are plain JSON, fully Git-friendly. Track them in version control to share prompts across your team.
Requirements
- VS Code 1.85.0 or later
- Azure OpenAI resource with deployed models (GPT-4, GPT-4o, and/or GPT-4o-mini)
- A valid API key and endpoint URL
License
MIT
