Spark Cell Runner
Run Databricks notebook-source files and SQL in VS Code — on Databricks, or fully offline on a local Spark pool.
An open source VS Code extension for Databricks engineers who want notebook-grade development — even when the serverless endpoint is unreachable.


Try it now — build and install in under a minute:
git clone https://github.com/Iamprashanth-1/spark-cell-runner.git
cd spark-cell-runner && npm install -g @vscode/vsce
npx vsce package && code --install-extension spark-cell-runner-0.7.0.vsix
Table of Contents
Why
Databricks notebook development usually means one of two things: a browser you
keep switching to, or databricks-connect that fails the moment a serverless
endpoint is unreachable or you're on a train.
Spark Cell Runner gives you a third option — a local, Jupyter-like
experience inside VS Code with two interchangeable backends:
|
Databricks mode |
Local pool mode |
| Engine |
Databricks serverless / cluster via databricks-connect |
Sail Spark Connect server on localhost |
| Network |
Cloud required |
Fully offline |
| Compute |
Real Databricks compute |
Single Rust process, ~1 s startup |
| Tables |
Real Unity Catalog |
Local Delta / Iceberg warehouse on disk |
Same notebook code, same compatibility shim, either backend.
How is this different from the official Databricks extension? The official
one connects you to a live Databricks workspace and runs cells on cloud
compute — the right tool for real jobs. Spark Cell Runner speaks the same
notebook-source format and shims the same dbutils APIs, but can run with
zero cloud dependencies on a local Sail pool, sync a slice of Unity
Catalog down for offline work, and execute ad-hoc SQL against a local Delta
warehouse. They coexist happily — use both.
Quick Start
1. Create and start a local pool (Configuration panel → Spark Pool → Create new pool):
[pool] "local" is serving on sc://127.0.0.1:59921
2. Open examples/01_hello_spark.py and run a cell — output appears on the cell and in the results panel:
Spark version: 4.2.0
+---+-------+
| id|squared|
+---+-------+
| 1| 1.0|
+---+-------+
3. Open a .sql file and press Ctrl+Enter — results stream into the output channel:
[sql] run selection - 1 statement(s) executed, 0 failed
[sql] 3 rows
[sql] id,label
[sql] 1,row-1
That's the whole loop: explore tables in the Unity Catalog panel → click
Query → edit → Ctrl+Enter → read results in the output channel.
Features
- Cell-level execution of Databricks notebook-source
.py files (and
.ipynb): run current cell, all cells, run-to-cursor, with per-cell
decorations, CodeLenses, and a results panel.
- Two connection modes — Databricks (serverless/cluster) or a local Spark
pool that needs no cloud access at all.
- Databricks compatibility shim —
%sql / %sh / %pip / %fs / %md
magics, %run expansion, dbutils proxies (widgets, secrets, fs, notebook,
library), /Workspace/ and /Volumes/ path translation.
- Local Spark pools — named Sail servers with manifests, logs, lifecycle
control, and isolated warehouses, managed entirely from the sidebar.
- Unity Catalog explorer — browse catalogs/schemas/tables/columns from
Databricks (REST) or the local warehouse (disk), with table previews and
per-table format labels (Delta / Iceberg).
- Unity Catalog sync — mirror dev schemas/tables into the local warehouse
(schema-only, or schema + data with a row limit), into either Delta or
Iceberg (
syncFormat).
- SQL execution — run
.sql files, selections, or statements on the pool
(Ctrl+Enter), with results in the output channel.
- Lakehouse container — expose the warehouse to Docker/Podman Desktop as a
browsable UI.
- SQL-aware environment picker — validates the interpreter for the active
mode (pysail + pyspark locally, databricks-connect in the cloud).
- pip-installable CLI — the same engine headless (
pip install spark-cell-runner): run notebook cells, SQL, UC sync, and pool management
from any terminal.
Architecture
VS Code (this extension)
┌────────────────────────────────────────────────────────────────┐
│ notebook editor / .py source Configuration tree commands │
│ │ │ │ │
│ ▼ ▼ │ │
│ scriptBuilder ──► session (JSON over stdio) ◄──── ──┘ │
│ │ │
│ session_driver.py (persistent python proc) │
│ │ bootstrap.py + runtime_data.py │
└──────────────────────────┼─────────────────────────────────────┘
│
┌────────────────────┴─────────────────────┐
▼ databricks-connect ▼ Spark Connect (sc://127.0.0.1:<port>)
Databricks serverless / cluster Local Sail pool (lakehq/sail, Rust engine)
(cloud, Unity Catalog) (offline, Delta warehouse on local disk)
│
┌─────────────┴──────────────┐
▼ ▼
~/.spark-cell-runner/warehouse Docker/Podman
(local Delta tables) stack (FileBrowser UI)
Run Pipeline
- Parsing (
parser.js) — notebook-source .py files are split into
cells on # COMMAND ---------- separators; # MAGIC lines are decoded
into magics (%sql, %pip, ...).
- Script building (
scriptBuilder.js) — each run compiles a flat
Python script: prior-cell context (or replay of executed cells),
translated magics, and expanded %run includes (parsed recursively and
cached).
- Session (
session.js + python/spark_cell_runner/session_driver.py) — one
long-lived Python process per notebook, speaking a JSON-line protocol over
stdio ({"type":"exec","id","code"} → {"type":"result","id","ok",...}).
State persists across cells, Jupyter-style. The session is keyed by
notebook URI and a config signature (interpreter, profile, cluster,
connection mode, pool) — changing any of those transparently recycles the
process. In local mode the session runs in the pool's environment, so the
interpreter may differ from the general pythonCommand setting.
- Bootstrap (
python/spark_cell_runner/bootstrap.py, assembled by pyResources.js) —
executed inside the session on first use. It creates spark/sql (either
DatabricksSession.builder... or
SparkSession.builder.remote("sc://127.0.0.1:<port>") depending on
connectionMode), installs the dbutils proxies, and monkeypatches
builtins.open / os.path so Databricks-style paths resolve to mapped
local folders (see workspacePathMappings).
- Results (
runner.js, ui/decorations.js, ui/resultPanel.js) —
output is captured per cell into decorations and a results webview; the
generated script and result text are written under tempFolder.
The session wire protocol (JSON lines over stdio) is deliberately tiny:
--> {"type": "exec", "id": 3, "code": "<base64 python>"}
<-- {"type": "ready"} (once, at startup)
<-- {"type": "result", "id": 3, "ok": true,
"stdout": "...", "stderr": "..."}
Because both sides of the connection live in the selected Python environment,
the protocol is the only coupling point — swapping the engine or the
connection mode never changes it.
Local Spark Pools
A pool is a detached Sail Spark Connect server
(python -m pysail spark server --ip 127.0.0.1 --port <port>) plus a JSON
manifest in ~/.spark-cell-runner/pools/<name>.json:
{
"name": "local",
"engine": "sail",
"pythonCommand": "C:\\...\\.venv\\Scripts\\python.exe",
"port": 51234,
"warehousePath": "C:\\Users\\<you>\\.spark-cell-runner\\warehouse\\local",
"pid": 12345,
"logFile": "...\\pools\\logs\\local.log"
}
- Manifests live in the user home, so pools are shared across workspaces and
survive VS Code restarts; the server process is detached from the
extension host.
poolManager.js handles creation (name validation, free-port selection,
warehouse creation), start (health-checked by probing the Spark Connect
port), stop (PID kill, taskkill /T /F on Windows), and per-pool logs.
- The server's working directory is pinned to its warehouse, so managed
tables land inside the pool's own folder.
- In local mode, notebook sessions deliberately run in the pool's own
Python environment — that's where the matching
pyspark client lives.
- A warehouse registry (
python/spark_cell_runner/warehouse_registry.py) re-registers
on-disk Delta tables into every session, so sail.<schema>.<table> named
queries resolve even though Sail's metastore is session-scoped.

Why Sail?
The local pool runs Sail (PyPI package
pysail) as the Spark engine:
- No JVM, no cluster — Sail is a single Rust process installed with
pip install pysail; a pool starts serving Spark Connect in about a second.
- Spark Connect protocol — notebook sessions are ordinary PySpark clients
(
SparkSession.builder.remote(...)), the same protocol Databricks uses, so
cell code stays engine-agnostic.
- Delta and Iceberg tables on local disk — Sail writes both open table
formats natively (pysail 0.4+) into the pool's warehouse directory; that's
what the explorer reads and what the lakehouse container browses.
- Swappable by design — the pool manifest records
engine, and the
bootstrap only needs a Spark Connect endpoint, so a future pool can point
at any Spark-Connect-compatible engine without touching notebook code.
Known limitation: Sail's metastore is session-scoped. CREATE SCHEMA
/ saveAsTable names are lost when the pool restarts — but the Delta files
are durable, and the warehouse registry + explorer + sync all read from
disk. For stable names, write to explicit paths
(df.write.format('delta').save('hello.db/mytable')).
Unity Catalog Explorer
A lazy tree view browsing catalogs/schemas/tables/columns from one of two
sources (switch with the swap button in the view title):
- Databricks — real Unity Catalog over the REST SDK (no compute needed).
Right-click a table → Sync This Table to pull just that table locally.
- Local warehouse — reads the pool's Delta files directly from disk,
so it always shows what actually exists, even when the pool is stopped.
- Auto (default) — local when the pool is running, Databricks otherwise.

Unity Catalog Sync
ucSync.js spawns python/spark_cell_runner/uc_sync.py as a one-shot process (deliberately
separate from notebook sessions). It reads metadata over REST with the
Databricks SDK and writes to the pool over Spark Connect, streaming JSON
progress lines back to the extension.
- schema mode — recreates catalogs/schemas and empty Delta tables with
identical columns/types. No cluster needed.
- data mode — additionally copies rows through a Databricks session
(honoring
useServerless/clusterId), with an optional row limit.
- Filters (catalog, schema, table glob, mode, limit, format) are workspace
settings, so syncs are reproducible per project.
The local warehouse speaks both open table formats natively — no JVM, no
Iceberg Spark jars, no conversion step. This requires pysail 0.4+ (the
extension installs the latest pysail when you create a pool).
|
Delta |
Iceberg |
| Write (Python) |
df.write.format('delta').save(path) |
df.write.format('iceberg').save(path) |
| Write (SQL) |
CREATE TABLE t (...) USING delta LOCATION '...' |
CREATE TABLE t (...) USING iceberg LOCATION '...' |
| Time travel |
VERSION AS OF 0 / option('versionAsOf', ...) |
option('snapshotId'\|'branch'\|'tag', ...) |
| On-disk marker |
_delta_log/ |
metadata/*.metadata.json |
Everything that touches the warehouse is format-aware:
- Warehouse registry — walks the warehouse and re-registers every on-disk
table (either format) into each fresh session, so
sail.<schema>.<table>
name queries resolve despite Sail's session-scoped metastore.
- Unity Catalog explorer — the tree shows a
delta or iceberg label per
table and reads the schema from each format's metadata on disk.
- Table previews — auto-detect the format from the directory layout.
- UC sync —
sparkCellRunner.syncFormat (delta | iceberg, default
delta) chooses the local format your Unity Catalog mirror is written in,
in both schema and data modes.
In SQL, use a file:/// URL for Iceberg LOCATION literals — SQL string
literals treat Windows backslashes as escapes. See
examples/06_local_iceberg.py for a runnable walkthrough.
SQL Execution
.sql files get CodeLenses ("Run on pool" per statement, "Run file" at
the top) and Ctrl+Enter / Shift+Enter keybindings.
- Select a statement → runs the selection; cursor in a statement → runs that
statement; results stream into the Spark Cell Runner output channel as CSV.
- A failing statement stops the run and reports its real error.
- Local tables are addressable by name (
sail.<schema>.<table>) thanks to
the warehouse registry, with a format-aware path-based fallback
(delta.\`` / iceberg.\``).
Lakehouse Container
containerManager.js detects Docker, then Podman, and generates a Compose
project spark-cell-runner-<pool> that bind-mounts the pool warehouse into
filebrowser/filebrowser. Compute stays native via the pool; no JVM images
are pulled. When the stack runs:
|
|
| Warehouse UI URL |
http://localhost:<port> (picked automatically, shown in the launch notification) |
| Username |
admin |
| Password |
Auto-generated on first launch — stored in <warehouse>/<pool>/.filebrowser-credentials.txt and re-applied on every launch |
Recent FileBrowser releases generate a random admin password on first boot
and require 12+ character passwords, so the extension owns the credentials
instead of relying on defaults. The stack listens on localhost only.
Data and State on Disk
~/.spark-cell-runner/
pools/
<name>.json Pool manifests (engine, env, port, warehouse, pid)
logs/<name>.log Sail server stdout/stderr per pool
warehouse/
<pool>/ Local Delta / Iceberg warehouse of that pool
docker-compose.yml Generated lakehouse container stack
.filebrowser.db FileBrowser login database (container stack)
.filebrowser-credentials.txt
Generated UI login, re-applied on every launch
<schema>.db/... Schemas and Delta / Iceberg tables written by notebooks
Generated scripts and result text per workspace go to
<workspace>/sparkCellRunner.tempFolder (default .spark-cell-runner/).
Configuration
| Setting |
Default |
Description |
pythonCommand |
python |
Interpreter (path or command) used for sessions, pools, and sync. |
tempFolder |
.spark-cell-runner |
Workspace folder for generated scripts and results. |
injectDatabricksBootstrap |
true |
Inject the bootstrap that creates spark/sql/dbutils. |
databricksProfile |
(empty) |
~/.databrickscfg profile for Databricks auth. Pick or create profiles from the Configuration panel (mirrors the Databricks extension: lists profiles via databricks auth profiles, or create one with OAuth / a PAT). |
clusterId |
(empty) |
Cluster ID for databricks-connect runs. |
useServerless |
false |
Use serverless instead of clusterId. |
connectionMode |
databricks |
databricks or local (Sail pool). |
localPool |
(empty) |
Name of the active local pool. |
ucExplorerSource |
auto |
Unity Catalog explorer source: auto, local, or databricks. |
workspacePathMappings |
{} |
Map /Workspace/...-style paths to local folders. |
secretValues |
{} |
Local replacements for dbutils.secrets.get, keyed scope/key. |
widgetValues |
{} |
Values for dbutils.widgets, editable in the sidebar. |
openPyFilesAsNotebook |
false |
Open .py files containing Databricks notebook markers (# Databricks notebook source or # COMMAND ----------) in the notebook view; other .py files stay in the normal editor. |
syncCatalog / syncSchema / syncTables |
(empty) |
UC sync filters (empty = all). |
syncMode |
schema |
schema or data. |
syncFormat |
delta |
Local table format written by UC sync: delta or iceberg (Sail writes both natively; pysail 0.4+). |
syncRowLimit |
0 |
Max rows per table in data mode (0 = unlimited). |
Commands
All commands live under the Spark Cell Runner category:
| Command |
Purpose |
Open as Databricks Notebook |
Open a .py source file in the notebook view. |
Run Current Cell / Run All Cells / Run Notebook to Current Cell |
Cell execution. |
Preview Current Cell Script |
Inspect the generated flat script. |
Show Cell Output |
Reopen the results panel for a cell. |
Restart Notebook Session |
Recycle the persistent Python process. |
Select Python Environment / Set Python Path / Use Active VS Code Interpreter |
Interpreter management. |
Set Connection Mode / Set Cluster ID / Toggle Serverless Mode / Set Databricks Profile |
Connection targets. |
Create Databricks Profile |
Add a new ~/.databrickscfg profile via OAuth (databricks auth login) or a personal access token. |
Edit ~/.databrickscfg |
Open the Databricks config file for manual profile editing. |
Create / Start / Stop / Delete Local Spark Pool, Show Pool Logs, Manage Local Spark Pool |
Pool lifecycle. |
Install Local Pool Packages |
Install pysail + pyspark-client into a venv you pick. |
Run SQL File on Pool / Run Selected SQL on Pool / Run Statement on Pool |
SQL execution (also Ctrl+Enter in .sql files). |
Sync Unity Catalog to Local Pool |
Schema/data sync from dev UC. |
Launch / Stop Lakehouse Container, Open Warehouse UI |
Docker/Podman stack. |
Query Table / Preview Data / Copy Name |
Unity Catalog explorer actions. |
Examples
Runnable sample notebooks live in examples/:
Open any of them with Spark Cell Runner: Open as Databricks Notebook and
run cell-by-cell.
Project Layout
src/
extension.js Entry point: activate()/deactivate() wiring only
constants.js Shared constants and magic-line regexes
state.js Shared mutable singletons (sessions, caches, sync result)
config.js sparkCellRunner configuration access + helpers
commands.js All vscode.commands registrations
parser.js Notebook-source parsing, cell splitting, magic handling
serializer.js Notebook <-> "# Databricks notebook source" .py serialization
codelens.js Run / preview CodeLenses in the text editor view
navigation.js Go to Definition / Find References remapping for cells
scriptBuilder.js Generated-script construction, %run expansion, magic translation
session.js Persistent per-notebook python sessions (JSON stdio protocol)
runner.js Run orchestration + notebook controllers
pythonEnv.js Interpreter discovery, Databricks Connect validation, child env
poolManager.js Local Spark pool manifests + Sail server lifecycle
sqlRunner.js SQL execution on the pool + result rendering
ucSync.js Unity Catalog -> local pool sync orchestration
containerManager.js Docker/Podman compose stack (FileBrowser over the warehouse)
pyResources.js Loads the Python templates and injects settings
ui/
configurationTree.js Configuration panel (native tree, Databricks-style)
unityCatalogTree.js Unity Catalog / warehouse explorer tree
sqlCodeLens.js Run-on-pool CodeLenses for .sql files
statusBar.js Status bar item (interpreter / profile / cluster)
decorations.js Per-cell run state decorations + run-state store
resultPanel.js Results webview (summary / stdout / stderr / script)
python/ The Python engine — a real pip package (see PyPI Package)
pyproject.toml Package metadata; publishes to PyPI as spark-cell-runner
spark_cell_runner/ Same modules the extension spawns (single source of truth)
bootstrap.py Session bootstrap: spark/sql, dbutils proxies, path translation
runtime_data.py Per-run widget/secret/path-mapping values (template)
session_driver.py Long-lived python process executing code blocks over stdio
uc_sync.py One-shot UC -> pool sync driver (JSON progress on stdout)
uc_explore.py Catalog/schema/table listing for the explorer (REST or Spark)
execute_sql.py Multi-statement SQL driver for the local pool
preview_table.py Format-aware Delta / Iceberg table preview driver
table_formats.py Shared Delta + Iceberg detection / walk / registration
warehouse_registry.py Backward-compatible facade over table_formats
parser.py Python port of parser.js for headless cell execution
bootstrap_builder.py Builds the connection preamble for the CLI run command
pool.py Pool lifecycle over the shared manifests (CLI twin of poolManager)
cli.py `spark-cell-runner` entry point (pool/run/sql/sync/explore/preview)
tests/ pytest suite for the pure modules
examples/ Runnable sample notebooks (see examples/README.md)
test/ Sample notebooks, UI previews (npm run preview:ui)
PyPI Package (pip install)
The whole Python engine ships as a pip package — the same modules the
extension spawns, published to PyPI as
spark-cell-runner:
pip install spark-cell-runner
The CLI is also installed as the short alias scr (scr pool create dev).
spark-cell-runner pool create dev && spark-cell-runner pool start dev
spark-cell-runner run my_notebook.py --pool dev
spark-cell-runner sql query.sql --pool dev
spark-cell-runner sync --catalog main --mode data --format iceberg
spark-cell-runner preview ~/.spark-cell-runner/warehouse/dev/demo.db/orders
What parity means: pools created from the CLI are visible in the extension
sidebar and vice versa (shared ~/.spark-cell-runner/pools manifests), the
warehouse layout is identical, and run executes the same Databricks
notebook-source format (%sql/%md magics included) with the same Delta +
Iceberg registry underneath. Databricks-mode runs need the extra:
pip install "spark-cell-runner[databricks]".
Build and Install
Requires Node.js and the vsce CLI:
npm install -g @vscode/vsce
npm run package # produces spark-cell-runner-0.7.0.vsix
Install (uninstall older versions first so cached assets refresh):
code --uninstall-extension PrashanthReddyMunagala.spark-cell-runner
code --install-extension spark-cell-runner-0.7.0.vsix
Then fully quit and restart VS Code. To uninstall:
code --uninstall-extension PrashanthReddyMunagala.spark-cell-runner.
No build/compile step is needed — the extension is plain CommonJS JavaScript
plus the Python engine in python/spark_cell_runner/, which vsce packages as-is.
Development and Testing
npm run preview:ui renders the configuration tree outline and results
panel to test/preview/ — iterate on UI with no extension reload.
- F5 launches an Extension Development Host on the
test/ workspace with
sample notebooks (widgets, %sql, %run).
- JS changes need an extension-host reload;
python/spark_cell_runner/*.py template
changes are picked up on the next run.
- Python resources must stay compatible with Python 3.10+ — check with
python -m compileall python/spark_cell_runner.
- See PUBLISHING.md for the Marketplace release flow.
Publishing to the Marketplace
One-time setup (publisher, PAT) and the per-release flow are documented in
PUBLISHING.md. In short:
npx vsce login PrashanthReddyMunagala # paste your Marketplace PAT once
npx vsce publish patch # bumps the version, packages, publishes
Or push a semver tag (e.g. v0.7.0) — CI packages, publishes, and attaches
the .vsix to a GitHub release (.github/workflows/release-vscode.yml,
needs the VSCE_PAT secret). The same tag also publishes the Python package
to PyPI via Trusted Publishing, so one tag releases both.
Publishing to PyPI
The Python package lives in python/ with its own pyproject.toml
(single source of truth with the VSIX: the extension ships the same
python/spark_cell_runner/ files inside itself).
cd python
python -m pip install --upgrade build twine
python -m build # sdist + wheel into python/dist/
twine check dist/*
twine upload dist/* # needs a PyPI token (see python/README.md)
Or let CI do it: push a semver tag (e.g. v0.7.0) — Trusted Publishing
authenticates the workflow via OIDC, no token secret needed
(.github/workflows/release-pypi.yml).
Bump the version in both python/spark_cell_runner/__init__.py and
python/pyproject.toml (keep them equal); the extension's own version stays
in package.json.
Troubleshooting
| Symptom |
Cause / fix |
SyntaxError: f-string: unmatched '(' at bootstrap |
Runner environment older than Python 3.12 with a bootstrap regression — fixed since 0.3.0. |
NameError: dcr_original_isfile at bootstrap |
Fixed since 0.3.0 (definition used before assignment). |
Local Spark pool bootstrap failed: No module named 'pyspark' |
Notebook sessions run in the pool's venv in local mode — start the pool via the sidebar; its env carries pyspark. |
No Delta tables found in <path> (explorer) |
Fixed since 0.5.5 (missing fs import). Also check the [uc-explorer] diagnostics line in the output channel. |
quay.io ... 401 Unauthorized when launching the lakehouse stack |
Old extension version using MinIO images; update — the stack now uses filebrowser/filebrowser. |
Named queries say Database not found after a pool restart |
Expected on old versions — since 0.6.3 the warehouse registry re-registers on-disk tables into every session. |
| SQL runs but results are missing |
Check the Spark Cell Runner output channel — results render there since 0.6.2. |
| Sync fails with SDK/auth errors |
Verify databricksProfile works (databricks auth login) and the pool is running. |
| The "open .py as notebook" toggle doesn't change how files open |
It writes workbench.editorAssociations (workspace, falling back to user scope). Remove any User-level *.py association and fully restart. |
Contributing
Issues and PRs are welcome. Keep in mind:
- Plain CommonJS JavaScript — no build/compile step.
- Python resources must stay Python 3.10+ compatible (no 3.12-only f-string
syntax); run
python -m compileall python/spark_cell_runner before submitting.
- Generated scripts and templates are validated as real Python — keep it that
way in PRs.
- Bump the version in
package.json (and the sidebar footer version string)
for every user-visible change.
License
MIT