Spark Studio — Local Apache Spark for Python & SQL

Install the extension, pick an engine, click Run. Real Apache Spark on your laptop — no cluster, no cloud account, no manual Java/Python wiring.

Every other Spark extension on the marketplace assumes a remote workspace (Databricks, Synapse/Fabric, HDInsight). Spark Studio is the one you install and have Spark — locally, immediately. Register a CSV as a table, run PySpark and Spark SQL from the editor, explore results in a sortable grid, and build Spark Declarative Pipelines — all from the VS Code sidebar.

How It Works

1. Pick an engine — once

On first activation, choose your engine from the Spark Studio sidebar or the walkthrough. Lite is ready in about a minute and needs no Java; Full Spark downloads a managed JVM for complete Spark semantics. You can switch anytime from the status bar.

Lite uses Sail, a Rust reimplementation of the Spark Connect protocol — your PySpark code runs unchanged. Full mode is the real Apache Spark distribution on a managed JVM. Nothing touches your system Python or system Java; it all lives in the extension's own storage.

2. Register a data file as a table

Right-click any CSV, Parquet, or JSON file in the Explorer → Register as Spark Table. It appears in the Tables panel, where you can expand it to see columns and types, preview its rows, or remove it — a mini data explorer in the sidebar.

3. Run Python and SQL from the editor

Open a .py or .sql file and click ▶ Run with Spark (editor toolbar, or the in-file CodeLens), or press Cmd/Ctrl+Enter. A Spark session is pre-wired — no SparkSession boilerplate, no spark-submit. SQL results open in a sortable grid; Python output (including df.show()) streams to the Spark Studio Run panel.

Everything runs in one persistent session, so a table you register in the sidebar is instantly visible to spark.sql(...) in your Python file, and views persist between runs like a REPL.

Features

• Zero-setup local Spark: the extension provisions its own Python venv, the engine, and (Full mode) a Temurin JRE — pinned versions, isolated storage, your machine stays clean
• Two engines, one click: Lite (Sail, no JVM) for instant SQL/DataFrame work; Full Spark (JVM) for 100% Spark semantics, MLlib, and pipelines. Switch from the status bar
• Run .py files with a pre-wired, shared Spark session — no imports, no spark-submit; run the whole file or just a selection
• Run .sql files / selections and view results in a sortable, themed grid
• Notebook kernel: open any .ipynb, pick the Spark Studio kernel, and run Python and SQL cells against the same session — variables and temp views persist cell-to-cell; SQL cells render as tables
• Sidebar data explorer: register CSV/Parquet/JSON files as tables, expand to inspect columns and types, preview rows, and remove tables — all from the Tables panel
• Spark Declarative Pipelines: scaffold, dry-run-validate, run, and visualize the dependency DAG of YAML + SQL pipelines (Full mode) — the open-source, local twin of Databricks Lakeflow/DLT
• Migration assistant: right-click a config-driven ETL YAML → convert it into a Declarative Pipeline project (one .sql per query), with runtime placeholders and self-references flagged for you
• Runtime Manager: a settings panel to switch engines and add feature packs (Pandas API on Spark, Delta Lake, ML helpers) with one Install & Restart click
• Snippets for PySpark (sp:) and Spark SQL (sql:) — sessions, reads/writes, joins, windows, MLlib pipelines, and declarative-pipeline tables
• Persistent REPL-style session: temp views and variables carry across runs; the sidebar, editor, and notebook share one Spark world

Scope

First-class support for Spark SQL, DataFrames, and MLlib in Python and SQL — the substrate of virtually all analytics-engineering and ETL work on Spark. Structured Streaming runs in Full mode. GraphX is intentionally out (it has no Python API). MLlib and streaming connectors run in Full Spark mode.

Installation

From the VS Code Marketplace (recommended) Search Spark Studio in the Extensions view and click Install.

Install from a .vsix Download spark-studio-<version>.vsix, then in the Extensions view: ··· → Install from VSIX…. Or from the terminal:

code --install-extension spark-studio-0.1.0.vsix

Build from source

git clone https://github.com/tripleaceme/spark-studio
cd spark-studio
npm install
npm run package        # produces spark-studio-<version>.vsix

Requirements

• Python 3.9+ on your PATH (or set sparkStudio.pythonPath). Everything else — the engine, Spark, pandas/pyarrow, and the Full-mode JVM — is downloaded and managed by the extension.
• Full mode: no manual Java install needed. Spark Studio provisions a Temurin JRE 17 automatically; point sparkStudio.javaHome at an existing Java 17+ if you'd rather use your own.
• ~150 MB free disk for Lite, ~500 MB for Full.

Extension Settings

Commands

Keyboard Shortcuts

Spark Declarative Pipelines

Full mode ships the spark-pipelines CLI. A pipeline is a folder with a small spark-pipeline.yml and SQL/Python transformation files; Spark infers the execution DAG from how tables reference each other:

my_pipeline/
├── spark-pipeline.yml       # name, catalog, where transforms live
└── transformations/
    ├── bronze.sql           # CREATE MATERIALIZED VIEW bronze AS ...
    └── silver.sql           # reads bronze — ordering inferred automatically

Right-click the YAML to Validate (dry-run) the whole graph without touching data, or Run it. Develop and test on your laptop, deploy the same files to a cluster.

How it's built

The extension is a pure orchestrator — it never bundles a JVM or native binaries. It manages a Python venv in its own storage, starts the engine (Sail server for Lite; a local JVM SparkSession for Full), and holds one persistent Spark session that the sidebar and editor share over a small JSON-lines protocol. Result data moves over Apache Arrow.

License

MIT © tripleaceme