Azure ML Model Drift & Health Dashboard

Proactive ML model health monitoring directly in VS Code. Detect data drift, track prediction quality, and get traffic-light alerts before your models silently degrade.

The Problem

After deploying ML models to Azure ML endpoints, there is no proactive VS Code-based monitoring for data drift, prediction drift, or performance degradation. Teams discover their model is broken when business metrics drop — weeks too late.

The Solution

This extension connects to your Azure ML workspace, pulls inference metrics in real-time, calculates statistical drift scores (PSI, KS-test), and presents a traffic-light dashboard right in VS Code.

"3 of your 7 models are degrading" — see it at a glance, not after a business review.

Features

Traffic-Light Sidebar

A dedicated Activity Bar panel shows all your online endpoints with color-coded health:

• 🟢 Green — PSI < 0.1, no issues
• 🟡 Yellow — PSI 0.1–0.2 or elevated latency
• 🔴 Red — PSI > 0.2, high error rate, or significant KS-test drift

Drift Dashboard (Webview)

Click any endpoint to see a detailed dashboard with:

• Overall PSI & KS scores
• Feature-level drift table with per-feature PSI, KS statistic, p-value, and visual drift bars
• Latency & error rate cards
• Request volume over the last 24 hours
• Active issues and deployment details

Status Bar Summary

A persistent status bar item shows at-a-glance health: ML: 5✓ 1⚠ 1✗

• Background turns red if any endpoint is critical
• Background turns yellow if any endpoint is in warning state

Proactive VS Code Alerts

When drift exceeds your configured thresholds, the extension:

• Fires a VS Code notification with a direct link to the dashboard
• Adds the alert to the Drift Alerts panel in the sidebar
• Deduplicates alerts (1-hour window per endpoint per metric)

Health Report Export

Generate a JSON health report for all endpoints — useful for MLOps reviews, compliance, and Slack/email integrations.

Getting Started

1. Install the Extension

Install from the VS Code Marketplace or from VSIX:

code --install-extension azure-ml-drift-monitor-1.0.0.vsix

2. Configure Your Azure ML Workspace

Open VS Code Settings (Ctrl+,) and search for mlDrift:

Or add to your settings.json:

{
  "mlDrift.subscriptionId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "mlDrift.resourceGroup": "my-ml-rg",
  "mlDrift.workspaceName": "my-ml-workspace",
  "mlDrift.refreshIntervalMinutes": 15
}

3. Connect

Open the Command Palette (Ctrl+Shift+P) and run:

ML Drift: Connect to Azure ML Workspace

The extension uses DefaultAzureCredential from @azure/identity, which supports:

• Azure CLI login (az login)
• VS Code Azure Account extension
• Managed Identity (in Azure-hosted environments)
• Environment variables (service principal)

4. Run a Drift Check

Right-click any endpoint in the sidebar and select Run Drift Check Now, or use:

ML Drift: Run Drift Check Now

Drift Detection Methods

Population Stability Index (PSI)

Compares the distribution of input features between a baseline (training) dataset and current inference data.

Kolmogorov-Smirnov (KS) Test

A non-parametric test measuring the maximum distance between two cumulative distribution functions.

Configurable Thresholds

Commands

Required Azure Permissions

The authenticated identity needs these Azure RBAC roles on the ML workspace:

Architecture

┌─────────────────────────────────────────────┐
│ VS Code Extension                           │
│                                             │
│  ┌─────────────┐  ┌──────────────────────┐  │
│  │ Tree View   │  │ Drift Dashboard      │  │
│  │ (Sidebar)   │  │ (Webview Panel)      │  │
│  │ ● Endpoint1 │  │ ┌──────┐ ┌────────┐ │  │
│  │ ● Endpoint2 │  │ │ PSI  │ │ Drift  │ │  │
│  │ ○ Endpoint3 │  │ │ Card │ │ Table  │ │  │
│  └─────────────┘  │ └──────┘ └────────┘ │  │
│                   └──────────────────────┘  │
│  ┌──────────────────────────────┐           │
│  │ Status Bar: ML: 5✓ 1⚠ 1✗   │           │
│  └──────────────────────────────┘           │
│                                             │
│  ┌───────────────┐  ┌────────────────────┐  │
│  │ Health        │  │ Drift Calculator   │  │
│  │ Monitor       │──│ (PSI, KS-test)     │  │
│  │ (Poll Loop)   │  └────────────────────┘  │
│  └───────┬───────┘                          │
│          │                                  │
└──────────┼──────────────────────────────────┘
           │
    ┌──────▼──────┐     ┌──────────────┐
    │ Azure ML    │     │ Azure Monitor│
    │ REST API    │     │ Metrics API  │
    └─────────────┘     └──────────────┘

Troubleshooting

License

MIT