CatalystOps — PySpark Optimizer

CatalystOps catches PySpark performance issues before they reach production. It runs 40+ anti-pattern checks instantly in the editor, validates schemas at edit time, analyzes Catalyst execution plans via Databricks dry run or SSH, inspects historical job runs from Spark event logs, and tracks Databricks spending in a built-in billing dashboard — without touching your data.

Install from the VS Code Marketplace

Features

Local Analysis — No Cluster Required

Runs instantly on every file open and save. Detects:

Streaming, Delta, and DLT pipelines have dedicated rule sets. Add # noqa: catalystops to suppress a line.

Hover cards — every issue shows a one-sentence explanation and a quick-fix code snippet on hover.

Schema Validation — No Cluster Required

When a StructType or DDL schema is defined in the same file, CatalystOps validates column references and types at edit time:

• Unknown column names with "did you mean?" suggestions
• Type mismatches (numeric/string/date/array functions on wrong column types)
• union(), intersect(), except() schema alignment — catches silent wrong-result bugs from column order mismatches
• Join condition type mismatches

Schemas propagate through .filter(), .select(), .drop(), .withColumn(), .withColumnRenamed(). External sources (spark.table(), spark.read without .schema()) are skipped — no false positives.

Static Cost Estimation — No Cluster Required

Annotate your file to get an instant dollar estimate via CodeLens:

# @compute: nodes=4, cores=2, memory=16GB, rate=0.25

events = spark.read.parquet("s3://bucket/events")  # @size: 50GB
lookup = spark.read.csv("s3://bucket/lookup")       # @size: 200MB

Dry Run — Catalyst Plan Inspection

Submits a neutralized version of your script to a Databricks cluster, serverless compute, or SSH tunnel. All writes and actions are replaced with explain("formatted") captures — no data is modified.

Plan issues detected: broadcast vs sort-merge vs cartesian joins, shuffle exchanges, too-few partitions, repeated table scans (including cross-DataFrame), cache spill, missing partition filters, missing table statistics, SinglePartitionBottleneck, SortAggregate, GlobalWindow. Fully Photon-aware.

Execution modes:

Job Run Analysis — Historical Runs

Analyze a past run directly from the Jobs sidebar — no re-execution needed. CatalystOps reads the Spark event log from DBFS, extracts physical plans, and opens an interactive DAG view showing operator trees, filter conditions in plain English, issue badges, and a View Source button.

Prerequisites: Cluster log delivery to DBFS must be enabled on the cluster. Serverless runs are not supported (no DBFS event log).

Clusters Sidebar — One-Click SSH

The Clusters panel lists all interactive clusters with their current state. Click the SSH icon on any cluster to:

1. Auto-start it if stopped (polls every 5 s)
2. Run databricks ssh setup automatically
3. Fix access mode / upgrade Spark if needed (one-click modal)
4. Open VS Code Remote SSH

Right-click a cluster for Stop Cluster and Reset SSH Host (clears cached alias).

Requirements: Databricks CLI ≥ 0.269, VS Code Remote SSH extension, DBR 17+.

Billing Dashboard

Queries system.billing.usage via Databricks SQL and shows spend by user, job, and workload type. Supports Day / Week / Month / Custom ranges. Results are cached for 1 hour; click Refresh to force a live query.

Requirement: Unity Catalog System Tables must be enabled on your workspace.

MCP Server — AI Assistant Integration

CatalystOps exposes live analysis data to Claude, GitHub Copilot (VS Code 1.99+), Cursor, and any MCP-compatible client via a Streamable HTTP server that starts automatically on a dynamic port.

Tools:

Resources: catalystops://issues/current, catalystops://plans/last, catalystops://billing/summary

Prompts: pyspark_code_review, optimize_spark_plan

VS Code Copilot discovers the server automatically. For other clients, add the URL shown in the CatalystOps Output panel:

{ "servers": { "catalystops": { "url": "http://127.0.0.1:<port>/mcp" } } }

Getting Started

1. Install — search for CatalystOps in the Extensions panel or:

ext install CatalystOps.catalystops

Local analysis works immediately — no configuration needed.

2. Connect to Databricks (for dry-run plan analysis) — run CatalystOps: Configure Databricks Connection from the Command Palette.

The wizard auto-detects available auth methods and shows only what applies to you:

Cluster ID is only prompted when running a dry-run — not during connection setup.

3. Analyze — open a .py file. Local checks run automatically. Press ⌘⇧K / Ctrl+Shift+K for a full dry-run plan analysis.

Commands

Settings Reference

Development

Prerequisites: Node.js v16+, VS Code v1.85+

git clone https://github.com/lezwon/CatalystOps
cd CatalystOps
npm install
npm run build

npm run compile   # TypeScript compilation
npm run watch     # Rebuild on save
npm run build     # Production bundle (esbuild)
npm test          # Run test suite

Press F5 in VS Code to launch an Extension Development Host.

License

Elastic License 2.0 (ELv2) — source is publicly available; hosting or redistributing as a competing service is not permitted.