ECIT Lakehouse Studio

A VS Code extension for developing Microsoft Fabric notebooks and exploring Fabric lakehouses. Everything runs on Fabric Spark via the Livy Sessions API; SQL tabs can additionally be flipped to the lakehouse's read-only T-SQL endpoint for instant exploration without a Spark session.

Note: This extension was originally built as internal tooling at ECIT and is now available for broader use. While it has been used in production internally, it has not yet been widely tested outside our environments. Issues are quickly resolved in frequent updates — feedback and bug reports are welcome.

Features

• Lakehouse Explorer — Browse schemas, tables, columns, and files. Create schemas and delete tables directly from the tree view
• Livy Spark Execution — Run notebooks and SQL on Fabric directly from VS Code
• SQL Intellisense — Auto-completion for table/column names in spark.sql() and .sparksql files, with cross-lakehouse support for 3-part naming (lakehouse.schema.table)
• T-SQL endpoint toggle — A status-bar toggle beside the Spark item (Spark SQL / T-SQL endpoint, shown on .sparksql tabs) runs SELECTs against the lakehouse's SQL analytics endpoint: instant, read-only, no Spark session, real T-SQL with [brackets]. Previews and Ctrl+3 follow it. Always starts on Spark SQL
• Refresh SQL endpoint metadata — The endpoint picks up lakehouse changes asynchronously, so a freshly created table reads as "Invalid object name" for a while. Right-click a connection (or a single table) to force the sync, and get a per-table report of what Fabric refreshed. Offered directly on that error
• Generate Script — Right-click a table → Generate script... → INSERT INTO / UPDATE / MERGE INTO / OPTIMIZE. Opens a new .sparksql tab prefilled with the right syntax and column list. Audit columns (e.g. ewhTimestamp) auto-excluded via the lakehouseStudio.scriptGenerationExcludeColumns setting. OPTIMIZE also comes with commented V-ORDER and VACUUM variants ready to uncomment
• DataFrame Viewer — View query results in a grid panel with Filter, Copy, and Excel export
• Notebook Development — Native Fabric .py notebook files serialize as real VS Code notebooks with cells, syntax highlighting, and remote execution — no conversion needed
• Fabric Compatibility — Supports %run, %pip install, notebookutils.notebook.run(), and notebookutils.notebook.runMultiple() via Livy
• Capacity Management — Monitor, resume, and pause Fabric capacity from the explorer
• Files Browser — Browse, upload, download, rename, and delete files in OneLake
• Multi-Connection — Run DEV and PRD sessions simultaneously with isolated connections
• MCP server — Register the bundled ecit-lakehouse MCP server in a repo with Lakehouse Studio: Setup ecit-lakehouse MCP server, so AI tools like Claude Code can query the lakehouse on the extension's own running Livy session (results pseudonymised by default)
• Git-Synced Workflow — Works with a git-synced Fabric workspace
• Authentication — Azure CLI / Interactive or Service Principal with secrets stored securely in VS Code
• AI-Assisted Development — Work with Claude Code, GitHub Copilot, or any VS Code AI tool directly alongside your Fabric notebooks — something not possible in the Fabric portal

Feature Demos

Quick Start

1. Install the extension from the VS Code marketplace
2. Prerequisites — install Python, Python packages, and Azure CLI (see Requirements below)
3. Open a folder containing Fabric .Notebook files (or any workspace)
4. Add a connection — click the "+" in the Lakehouse Studio sidebar
5. Start a Spark session — click "Spark" in the status bar (lower left)
6. Build intellisense — expand the Tables section in the Lakehouse Explorer to initiate schema sync
7. Run queries — press F5 in a .sparksql file or Ctrl+Enter in a notebook cell
8. Recommended settings — see Recommended VS Code Settings for the best experience

Requirements

Install Azure CLI (if not already installed):

winget install -e --id Microsoft.AzureCLI

Python packages (for intellisense only, not execution):

python -m pip install pyspark==3.5.1 requests pytz pandas openpyxl

Authentication

Two authentication methods are supported. In both cases, the user or service principal must have at least Contributor role on the Fabric workspace(s) used in the connection.

Azure CLI / Interactive (recommended)

Uses DefaultAzureCredential from @azure/identity, which tries Azure CLI, environment variables, and managed identity in order. Best for individual developers.

1. Run az login in your terminal
2. When adding a connection, select "Azure CLI / Interactive"
3. Only Workspace ID and Lakehouse ID are required

Tip: If the interactive login popup doesn't appear or gets stuck, run az login manually in the VS Code Terminal, then switch back to the Lakehouse Data Viewer pane.

Service Principal

Uses ClientSecretCredential for automated scenarios or shared team environments.

1. Create an App Registration in Azure Entra ID
2. Grant it Contributor access to the Fabric workspace
3. When adding a connection, select "Service Principal" and enter Tenant ID, Client ID, and Client Secret

The client secret is stored securely in VS Code's secret storage.

Connection Setup

Click Add Connection in the Lakehouse Studio sidebar:

Workspace Structure

The extension creates a local_development/ folder in your workspace root (add it to .gitignore — it holds machine-local state, including which Azure account you last signed in as):

your-repo/
├── local_development/                    # Auto-created, git-ignored
│   ├── active_connection.json
│   ├── workspace_imports/                # Auto-generated Python modules
│   └── connections/
│       └── {uuid}/
│           └── schema_reference.json     # Table/column metadata
├── *.Notebook/                           # Fabric notebook folders
│   └── notebook-content.py
└── utility/                              # Shared Python modules (optional)

The extension auto-discovers .Notebook folders anywhere in your workspace. Override with the lakehouseStudio.notebookSearchPath setting.

Key Commands

Spark SQL Files (.sparksql)

Standalone SQL files with SSMS-like experience:

• Syntax highlighting for Spark SQL
• Intellisense for tables, columns, and functions
• Press F5 to execute and see results in DataFrame Viewer
• Multiple statements separated by semicolons (;)
• T-SQL endpoint: click the Spark SQL status-bar item to switch to T-SQL endpoint — SELECT/WITH only, instant, no Spark session; column names are inserted as [brackets]. Requires the connection's SQL endpoint (looked up automatically and shown as "SQL Endpoint (auto)" in the connection dialog). The toggle resets to Spark SQL on every restart

SQL Cells in Notebooks (%%sql)

Write SQL directly in notebook cells:

%%sql
SELECT
    a.customer_id
    , a.name
FROM bronze.customer a
WHERE a.active = true

Type %%sql on the first line → full SQL highlighting and intellisense.

DataFrame Viewer

Results are capped at 1000 rows. The "123" toggle in the panel title bar turns on a background total-row count (Showing 1000 of N rows) for subsequent queries.

Results are cached per .sparksql tab for instant switching.

Settings

{
  "editor.autoIndent": "keep",
  "workbench.editor.enablePreview": false,
  "python.analysis.autoIndent": false,
  "editor.autoIndentOnPaste": false
}

Capacity Management

Configure capacity fields (Subscription ID, Capacity Name, Resource Group) on a connection to enable:

• Status indicator — green (Active), orange (Paused/Resuming)
• Resume/Pause — right-click the connection in the explorer

Required permissions: The user (interactive) or service principal must have Contributor role on the Fabric capacity resource in Azure (not the Fabric workspace — the Azure resource Microsoft.Fabric/capacities).

Changelog

2.6.3

• New MCP tool: start_spark_session. Starts the Spark session through VS Code as an explicit, visible step (default environment, with progress and Stop-button cancel), and just reports the session when one is already running. Nothing requires it — query, execute and run_notebook keep starting a session on their own — but an agent that wants to warm the session before a batch, or make the start a separate step, now has a tool for it
• Fixed: connection_info made the agent believe Spark was off. The tool only looks and never starts a session, which is right, but its "not attached" line quoted the refusal text ending in "start it in VS Code" — so an agent that called connection_info first concluded no session was possible, worked around it for the rest of the task, and reported afterwards that Spark was not on. The line now says the state is about now, that the tool started nothing, and which call starts one; the tool description and the server instructions say the same

2.6.2

• Fixed: an MCP server pinned to PRD could not use the Spark session you already had running. The extension runs ONE session and retargets it to another lakehouse by rewriting environment variables, so a session started on DEV keeps its DEV address — and the PRD server, which only ever looked at the PRD address, reported "no running Spark session" while VS Code was querying PRD perfectly well in that very session. Discovery now searches every connection this repo has, your own connection first, and attaches to whichever one holds your session. The session is still addressed where it lives; each statement pins your server's own workspace and lakehouse, so a bare schema.table resolves against the connection the server is pinned to, exactly as before
• Notebook runs are pinned the same way now. They were not pinned at all before, so a notebook run through the MCP built its paths from whatever lakehouse the session was created against — right when the session happened to match, wrong and silent when it did not. The workspace and lakehouse IDs and KEY_VAULT_NAME are set for the run and restored afterwards
• New: the MCP server asks VS Code to start Spark instead of just refusing. With no session running, the server sends a start request to the extension, which starts one on the default environment (no environment picker — there is nobody watching this window to answer it) and shows a toast saying Claude Code asked for it. The agent then just runs your query. Turn it off with LAKEHOUSE_MCP_AUTOSTART=0 in the server's env block in .mcp.json
• When the start cannot happen, the agent is told which of the three reasons it was and to stop rather than retry: VS Code is not open on this repo, the Fabric capacity refused (its compute limit is reached, or it is paused), or the login is wrong. The capacity reason is new information — Fabric puts it in a field nothing read before, so a failed session start used to say only "dead"; it is now in the VS Code error toast as well
• Fixed: Ctrl+Shift+R did nothing inside a Fabric notebook. It was scoped to SQL tabs only. It now works with a notebook focused, and refreshes the columns for every table the WHOLE notebook references rather than just the cell you are in
• Fixed: a dropped schema stayed in the Lakehouse Explorer tree. Two causes, both fixed: the schema list was only written to disk when a table had been added or removed, so dropping a schema whose tables were not cached never reached the tree; and DROP SCHEMA ... CASCADE can leave the schema's now-empty folder in OneLake, which is what the schema list is built from. The empty folder is now removed after the drop, and a folder that still holds something is reported instead of being silently deleted
• Fixed: the "set up the MCP server" offer never appeared for a colleague on a fresh clone. The offer needs the repo to have connections, but a fresh clone has none until you add one — and it was only ever made at startup, before that could have happened. It is now also made when you save a connection, which is the moment it becomes possible

2.6.1

• Fixed: MCP calls randomly failed with Fabric API unreachable: [WinError 10061] ... actively refused. A refused TCP connect happens before the request leaves the machine, and the server had no retry at all, so one unlucky socket killed the whole query or run_notebook. urllib opens a fresh connection per request with no keep-alive, so a 20-minute notebook run was hundreds of independent chances to hit it. Connect-phase failures (refused connect, DNS) are now retried twice with a short backoff
• The retry is deliberately narrow: only errors that prove nothing was sent, which is what makes it safe to retry a Livy statement submit without any risk of running SQL twice. Connection resets and timeouts — where the request may already have been sent — are not retried, and an HTTP 500 is never retried either
• Investigated with a 2,200-request repro against a live dev lakehouse: zero refused connects even at 137 connections/second, so the cause is machine-local (security agent, VPN) rather than anything in Fabric. The retry is the right fix regardless — it turns a random hard failure into an invisible hiccup
• Fixed: a Fabric API throttle killed notebook runs that were still going. The same repro tripped a real HTTP 429 after ~150 requests in ~11 seconds, with Retry-After: 300. Nothing handled 429, so a throttle landing mid-run aborted the poll and orphaned a statement that was still running on Fabric — the work continued, the result was lost. The poll now waits the throttle out and keeps going; the statement timeout still applies, so it can never run longer than you allowed
• Fixed: a throttle reported itself as a blank error. Fabric sends 429 with an empty body, and every error message pastes the body in, so the whole explanation read (HTTP 429): and stopped. It now says it was throttled, for how long, and that the limit is per caller rather than per workspace
• If a run does time out after being throttled, the message now says how much of the time was spent waiting rather than running, so it is not mistaken for a slow query

2.6.0

• New MCP tool query_multiple — run up to 10 independent read-only queries in ONE call, five at a time inside a single Spark statement. The agent was calling query five times in a row constantly; measured on a dev lakehouse, five separate calls take 6.2-7.1s against 1.7-2.5s for the batch. One bad query comes back as an inline error beside the results that worked, instead of a failed call
• Reads only, deliberately: the queries run concurrently with no defined order, so a CREATE TEMP VIEW and the SELECT that reads it cannot be batched — those are refused by index, naming query/execute as the place for them. Each result keeps its own footer, redaction and truncation note, and a result that hits the batch's per-query collection cap says so
• New MCP tool run_notebook_multiple — run several of the repo's notebooks as one DAG (default concurrency 5) and get a per-notebook outcome for each: ok, failed with its error, or skipped because a dependency failed. The result spells out the call that re-runs exactly the ones that failed, which is the loop this exists for: run a batch, fix the two that broke, re-run those
• It runs through the same notebookutils.notebook.runMultiple shim the customer notebooks already use, so topological waves, cycle detection and continue_on_error behave identically. dependencies orders the run; params overrides PARAMETERS-cell variables for every notebook and args for one. A notebook that exits reporting its own failure is counted as failed, not ok
• Duplicates, cycles, self-dependencies and any dependency naming a notebook outside the batch are refused before anything runs — the shim ignores the last of those silently, which is the kind of wrong that looks like it worked
• Fixed: the MCP stop button cancelled the wrong statement. The server tracked one in-flight statement, so with two tool calls running it cancelled whichever had submitted last. It now tracks each call separately and cancels the one you stopped
• Fixed: the Spark spinner stopped too early. With two MCP statements running, the first to finish deleted the activity file the extension watches, so the status bar went idle while the other was still working. The file is now removed only when nothing is in flight, and written atomically so the extension can never read it half-written
• A notebook that a batch member runs itself now appears in the batch report too, and a failure there fails the batch — the same notebook could otherwise be reported failed by run_notebook and OK by run_notebook_multiple
• run_notebook's dag_concurrency parameter is deprecated and still ignored; use run_notebook_multiple, whose concurrency is real
• High Concurrency Livy sessions were re-implemented and then abandoned during this release. The reasoning and the measurements are in CLAUDE.md so the question does not come round a third time: extra REPLs add Python interpreters, not Spark executors, and they fragment the temp-view namespace — parallelism belongs inside the session, which is what the two tools above do

2.5.8

• The wrong-identity warning no longer fires about tenants — only about the signed-in az USER. It used to compare the Azure CLI's active tenant against the connection's, but the active tenant decides nothing: every token is minted with --tenant <connection.tenantId> and az account get-access-token mints for any tenant already in the CLI's cache. Measured on a real machine — 22 connections across 10 tenants, one az context — the comparison was true for 9 of 10 connections at all times while every call worked fine
• It was worse than noise, because of what the button did: Sign in ends the Spark session and flips the machine's active tenant, so a second VS Code window on another customer toasted next, killed its own session and flipped back — ping-pong across two windows over tokens that were never wrong
• The tenant check also masked the one that matters: the user axis was gated behind it, so two accounts in the same tenant (the adm.* pattern) produced the less useful message. That axis now runs unconditionally, and its dedupe key dropped the tenant with it
• Nothing is lost. A tenant your account genuinely cannot use still fails the token request, and the existing "Authentication failed — you may be logged into the wrong tenant" toast with its Login... button already covers that — on a real failure instead of on a guess
• The connection dialog's Tenant ID hint, and four code comments, no longer describe a warning that compared tenants; the CLAUDE.md claim that Workspace Git Sync "depends on az login being on the right tenant" is corrected too — it pins the tenant like everything else and depends on the signed-in user, whose Git credentials Fabric has registered

2.5.7

• The Spark SQL / T-SQL endpoint toggle no longer hides itself. It appears whenever the active connection has a SQL analytics endpoint, instead of only while a .sparksql tab is focused. The toggle was always global and the tree's SELECT top 1000 rows and Ctrl+3 always followed it — but the only way to set it was a control you could not see without a SQL tab open, so previewing a table over T-SQL meant starting Spark, running the preview, flipping the toggle in the tab it opened, and running it again. Now: flip it, right-click the table, and the rows come back off the endpoint with no Spark session at all
• The tooltip now names what the toggle governs (SQL tabs, "SELECT top 1000 rows", Ctrl+3) and what it never touches (notebook %%sql cells and Python spark.sql() blocks, which always run on Spark) — worth saying now that the item is visible while a Python file is on screen
• Deleting the active connection, and the automatic prd→dev switch at startup, now re-render the toggle for the connection you land on (and drop a T-SQL selection that the new connection cannot honour)

2.5.6

• Refresh SQL endpoint metadata. Right-click a connection and pick Refresh SQL endpoint metadata (last in the menu) to make the lakehouse's SQL analytics endpoint pick up recent table and column changes. The endpoint syncs asynchronously, which is why a table Spark just created reads as "Invalid object name" over T-SQL for a while. A progress toast ticks the elapsed time, and the completion toast reports what Fabric actually decided per table — 42 table(s) refreshed, or a warning naming the failures with their error codes and a Copy details button. Also on table rows ("Refresh SQL endpoint for this table"), which scopes the call to one table and finishes in seconds
• Offered exactly when you need it: a T-SQL query that fails with "Invalid object name" or "Invalid column name" now follows the error with a short explanation of the async sync and a Refresh SQL endpoint button
• Two commands that had gone missing came back — both were left contributed but unregistered by the 2.5.5 engine removal, so they threw "command not found": the $(history) query history picker in the Results panel title bar (now also showing which engine each query ran on), and the table right-click Copy to another Lakehouse...
• "Refresh lakehouse table schemas" now refreshes the connection you clicked. On an inactive connection it silently synced the active one instead and reported success; the same bug in the per-schema refresh could write another lakehouse's schema into the active cache
• Dead-end warnings now carry the fix. "No Spark session" toasts have a Start Spark button (and, on SQL tabs whose connection has a SQL endpoint, Use T-SQL endpoint, which switches and re-runs the query straight away). "This table belongs to an inactive connection" offers Switch to this connection. Cancelling the environment picker on Tree Play no longer reports a missing session
• Ctrl+Shift+R no longer shadows Refactor. The column-refresh shortcut was bound with no context at all, so it replaced VS Code's Refactor command in every file and every language; it is now limited to SQL tabs
• MCP server 0.7.3 — it will not run your queries in a colleague's Spark session. When you had no session of your own and a teammate did, the server attached to theirs: your agent's statements queued behind their work, in their temp views and on their idle clock. It now refuses, naming who holds the session, with LAKEHOUSE_MCP_ANY_SESSION=1 as an opt-in for shared-session work
• A connection whose SQL endpoint id was never stored (anything saved between 2.3.0 and 2.5.5) now back-fills it instead of staying stuck without one

2.5.5

• The Local engine is gone. Fabric Spark via Livy is the only execution engine again — for F5, previews, notebooks, Tree Play and the MCP server. The team judged Spark adequate and performance less critical than the second engine's cost. Removed: the Setup local engine / Uninstall local engine / Local engine: start / stop / restart commands, the Local engine status-bar item and notebook kernel, the three lakehouseStudio.aileron.* settings, and the MCP server's local backend (server 0.7.2, Livy-only).
• One-off cleanup at first start: the engine package is uninstalled from the shared Python environment silently (nothing happens on machines that never had it), followed by a single "Local Engine uninstalled - please use Spark only in the future" toast. If another process holds the environment, its processes are stopped and the uninstall retried; a still-failing removal retries at the next start. The environment itself is kept — the MCP server runs from it.
• T-SQL endpoint toggle for SQL tabs. A status-bar item right of the Spark item — Spark SQL / T-SQL endpoint, visible on .sparksql tabs whose connection has a SQL analytics endpoint — flips ALL SQL tabs to the lakehouse's read-only T-SQL endpoint: instant SELECTs with no Spark session, real T-SQL (TOP, [brackets]; completions, alias.* expansion and the tree's "SELECT top 1000 rows" insert bracketed columns), 1000-row cap with the optional total count, Escape cancels. Non-SELECT statements are refused with a pointer back to Spark SQL. Tree previews and Ctrl+3 follow the toggle. It is global, resets to Spark SQL on every start, and never applies to notebook %%sql cells or Python spark.sql() blocks.
• Connections know their SQL endpoint. Looked up from the Fabric API on save (read-only "SQL Endpoint (auto)" under "Lakehouse Name (auto)" in the dialog) and back-filled for every connection of the open repo at start, in one settings write.
• Spark environment: always asked, "Default environment" preselected. When the workspace has environments, every start (status bar, F5, notebook cell, Tree Play) shows the picker with Default environment (the Fabric starter pool — fastest start) preselected, so light/small environments are a per-start choice. Nothing is remembered between starts; the old saved-environment pin is gone. Workspaces with no environments start silently as before.

2.5.4

• The prd allow-list now names TOOLS, and understands *. LAKEHOUSE_MCP_ALLOW took capability nouns (notebooks, pipelines) while the things it gates are tools, could not express "everything", and could not enable execute at all. The vocabulary is now execute, run_notebook, run_pipeline and *; the old spellings still work. Setup writes per-environment defaults: a prd entry gets run_notebook,run_pipeline (re-running last night's failed load is ordinary work — those are reviewed, in git; ad-hoc SQL is not), and a dev entry gets *. A fully-open prd server is now reachable, by a deliberate hand-edit
• Existing .mcp.json entries now pick up new settings automatically. The activation updater installed the new server package but never touched the entries, so an entry written before v2.5.3 had no LAKEHOUSE_MCP_ALLOW key and a prd server silently kept refusing the run tools its new default is meant to permit — which looks like the update deleted the key, when it was never there. Missing keys are now filled in with the per-environment default; values already present are left alone, so a deliberate hand-edit survives an extension update. Setup is still how you reset to defaults, and ALLOW: "" is how you opt out (an empty value is a value, a deleted key gets re-filled)
• What * can never reach: semantic-model deploys. They are absent from the vocabulary, so no value of the variable unlocks them — the deploy target is a workspace the connection cannot vouch for, and prd publishing goes through the release pipeline. It also stays an allow-list rather than the friendlier-reading deny-list, so a typo costs access instead of silently granting production writes. Refusals now carry the exact .mcp.json edit that would allow the tool, and --read-only overrides the list entirely
• "Login to Azure..." now actually switches identity. Some solutions use a different account in the same tenant (an adm. user, say). Because every check was tenant-based and a tenant-pinned token for your normal user mints fine, nothing noticed — Fabric just answered 403 / "not found" / empty session lists, which looks like a broken workspace. Signing in now stops any running engine (the Spark session and the Local engine kernel, no confirmation — a Livy session's identity is fixed by Fabric when it is created, so it can never become the new user), drops every cached token and credential, and records which az account you signed in as. Start an engine again and it runs as that account. No window reload needed
• A warning when the CLI is signed in as someone else. At activation and on connection switch, the az account is compared against the one your last login to that connection used, and a mismatch offers Sign in — the same toast that already existed for a wrong tenant, now covering the wrong user in the right tenant. The connection tooltip shows Signed in as …
• The MCP server picks up a re-login without restarting. Claude's queries used to keep the previous identity for up to an hour (a valid, unexpired token has nothing to trigger a refresh). The server now watches the recorded login and re-mints on the next call. connection_info also reports az user and extension last signed in as, with an explicit AUTH WARNING when they differ — so a permissions-shaped failure names its real cause
• MCP server bumped to 0.7.1 — the extension updates colleagues' installs at activation as usual (Reload Window to pick it up). The update check is a version equality test, so the two changes above share one bump deliberately: a single hop from 0.6.0 delivers both

2.5.3

• Notebook and pipeline runs can be selectively allowed on read-only (prd) MCP servers. New LAKEHOUSE_MCP_ALLOW env var in the server's .mcp.json entry — a comma list of notebooks and/or pipelines that re-enables run_notebook/run_pipeline on a prd connection. Deliberately an allow-list, not a read-only off-switch: SQL writes/DDL and semantic deploys can never be unlocked on a read-only server. Default is empty (exactly the old behavior), Setup writes the key so it is discoverable, and re-running Setup resets it to the safe default — same convention as LAKEHOUSE_MCP_REDACT
• run_notebook results now name the notebook in the headline (OK · run_notebook nb_gold_sales · engine … · connection …), so a collapsed tool result in Claude Code shows what ran without expanding it
• "Select..." buttons for the capacity fields in the connection dialog. Azure Subscription is picked from the subscriptions your login can see, and picking a Fabric capacity fills both the capacity name and its resource group in one go (listed via ARM from the subscription). Same auth as the workspace/lakehouse pickers; note that a service-principal connection only sees subscriptions/capacities where the SP has Azure RBAC (e.g. Reader) — interactive (Azure CLI) is the happy path
• MCP server bumped to 0.6.0 — the extension updates colleagues' installs at activation as usual (Reload Window to pick it up)

2.5.2

• New MCP tool job_history — recent runs of a pipeline (or notebook) in the orchestration workspace: status, start/end, duration, how it was invoked, and the failure reason for failed runs, newest first with the expected-runtime line. Answers "did last night's run succeed" and is the first stop when debugging a failing pipeline — WITHOUT starting a new run. Read-only, needs no engine, works on prd connections; failed rows carry the job instance id that run_pipeline accepts to resume watching
• New MCP tool list_pipelines — the repo's data pipelines with their declared parameters, plus its notebook names (exactly what run_pipeline/run_notebook accept). Purely local: no network, no engine, no login; pipeline names no longer have to be discovered by making a call fail
• MCP server bumped to 0.5.0 — the extension updates colleagues' installs at activation as usual (Reload Window to pick it up)
• SQL intellisense from an open .sparksql tab and an open Python file no longer fight each other. Temp views are now tracked per document, so defining a view in one file type no longer silently breaks completions and validation for views defined in the other — both resolve everywhere, deterministically
• Closing a Python file now clears its SQL squiggles from the Problems panel (they used to stick around until reload)
• One SQL provider implementation for both languages. The completion, diagnostics and code-action providers for Python spark.sql() blocks and .sparksql files were near-identical copies; they are now single classes with the "where does the SQL live" difference isolated in sqlContextSource.ts (~1,000 duplicated lines removed). Along the way: both languages share the union keyword list, Python gained completions inside single-line and f-string spark.sql(...) calls, and .sparksql "all columns" suggestions now scope to the statement at the cursor instead of the whole file
• Queued notebooks now honour notebookutils.notebook.exit(...). The tree-Play queue used to turn an exit call into a no-op, so cells after an early-exit guard still ran and the notebook reported success. An exit now stops the remaining cells and the notebook counts as completed — matching what the same notebook does in Fabric. Works on both engines
• A queue that fails to start no longer leaves the Results panel stuck in sequence mode
• Dead-code cleanup: removed the never-called Fabric notebook/folder REST methods (listFolders, createNotebook, getNotebookDefinition, …), five dead exports, unused imports, and the untouched test scaffold (src/test/, .vscode-test.mjs, three npm scripts and their dev dependencies). No behavior change

2.5.1

• New: "Create feature workspace from current branch..." on a connection's right-click menu. One click builds what the Fabric portal's branch-out does: a workspace named exactly after your current git branch, on the base workspace's capacity, connected to Azure DevOps on that branch, fully synced from Git, and related to the orchestration workspace via Fabric's new (preview) Workspace Relations API — so the portal shows the workspace tree and breadcrumbs back to the base
• The branch must already exist and be pushed — the command never creates branches. It refuses with a specific message on main/master, a detached HEAD, a branch that doesn't exist on origin (never pushed — or already merged and deleted while still checked out locally), a production connection, a connection without a Notebook Workspace ID, or a paused capacity (checked up front so you get "resume it first" instead of a cryptic API error)
• Re-running resumes instead of duplicating. A failure mid-chain leaves a real workspace behind; the command probes each step (workspace by name, git connection state) and continues from where it stopped. A same-named workspace connected to a different branch or repo is refused as a genuine collision
• The progress toast narrates the slow part — "Syncing items from Git — can take 10+ min" with a ticking elapsed time, so the wait never looks hung. That first sync really does take ten minutes or more, so the wait is watched for up to an hour — and if it somehow runs longer, you get an honest "still syncing, Fabric is continuing in the background" rather than a failure, because nothing has failed. One confirmation modal before anything is created, naming the workspace, base, capacity and branch
• New: "Clean up feature workspaces..." — lists the base workspace's branch-related workspaces, checks each one's branch against origin in a single git ls-remote, and pre-selects the ones whose branch was merged/deleted. Multi-pick, one confirm, then deletes the Fabric workspaces — git branches are never touched
• All calls in this feature run as your az CLI identity (same as workspace Git sync): Fabric's git-connect with automatic credentials is blocked for service principals, and a feature workspace should be attributed to the developer who made it. Notebooks in the feature workspace keep pointing at the shared dev lakehouse — nothing is rebound

2.5.0

• deploy_semantic and refresh_semantic now deploy your OWN copy of the semantic model, where the solution works that way. If the dev semantic workspace contains a personal_models folder, both tools target <Model>_<YourUsername> inside it — the same username the extension puts on your Livy sessions — and the model the whole team shares is left alone, because it is deployed by the pipeline that runs on main. That is the point: several developers deploying to one shared model overwrite each other's work

• Nothing selects this and nothing can override it. The folder's existence in the workspace is the entire switch, so a solution opts in by creating the folder (Fabric UI or setup script) and every deploy follows. Solutions without the folder behave exactly as before, byte for byte

• Your first deploy just works — it creates your personal model and does not need allow_create. That flag still guards creating a shared model, where a mis-aimed deploy would leave a rogue item behind

• If the check can't run, the deploy refuses. Not knowing whether a workspace uses personal models means not knowing whether the target is your copy or the team's, so a failed folder lookup is a refusal that names the failure, never a quiet fall back to the shared model

• The result names the model that was actually published, and says the shared one was not touched — so there is no guessing which item changed

• New check_semantic tool: a read-only preflight that reports the pinned workspace, whether it uses personal models, the exact name a deploy would publish under, and whether it exists yet. Publishes nothing

• Results now name the workspace, not just its GUID — -> workspace ecit-dpf-insights-dev with the id on its own line. The semantic workspace is pinned as a GUID nobody can eyeball, so every result used to send you off to look up which "insights" workspace it meant. Best-effort: if the name can't be read you get the GUID exactly as before, because a missing label must never block a deploy

• …and they hand over ready-to-paste connect parameters for the Power BI modeling MCP server, so picking the right model later is a copy, not a hunt:

  ConnectFabric: workspaceName='ecit-dpf-insights-dev'
                 semanticModelName='sem_ecitdataai_EmilLudvigsen'
  

  Both fields are exact-match on that side, and neither is derivable from a GUID — in personal mode the model isn't even the name in the repo. Printed by check_semantic and by a successful deploy/refresh; withheld when the model doesn't exist yet or the workspace name couldn't be read, since half a pair of exact-match parameters just invites a guess at the other half

• refresh_semantic refuses rather than refreshing the shared model when your personal copy doesn't exist yet, and points at deploy_semantic

• Semantic-model lookups are now paged. Personal models multiply the item count by the number of developers, and a first-page-only list would report an existing model as missing — which on the deploy path creates a duplicate instead of updating it

• Fixed: the MCP server told Claude it would fall back to the local engine when no Spark session was running. That fallback was deleted in v2.4.0 — both backends attach to something the extension owns, so "the engine you asked for isn't running" has one honest answer — but the server's own instructions kept promising it for two releases, which is the kind of stale text that actively misleads rather than merely ages. They now state the real contract: an engine must be running, there is no fallback between them, and if the active one is down the tool refuses and names the status-bar item that starts it. Claude is told to relay that and stop rather than retry or reach for the other engine, then try again once you say it's up. A selftest assertion keeps the old promise from coming back

• When the MCP server auto-updates, the toast now offers "Reload Window". pip replacing the package does not replace the running server process, so the update used to be announced and then quietly not in effect until Claude Code happened to restart — which for this release would mean still deploying to the shared model. It stays a dismissible toast (it fires at startup, so it must not throw away editor state), and it names the terminal case separately, because a Claude Code running in an integrated terminal can survive a window reload

• Files touched: python/mcp-server/ecit_lakehouse_mcp/{fabric_jobs,gate,smrun,tools,server,selftest,__init__}.py, python/tests/test_mcp_smrun_personal.py, src/setupMcpServer.ts, package.json, README.md, CLAUDE.md

2.4.3

• "Deploy…" is now "Deploy from DEV to PRD…", and it sits on every connection — active or not, dev or prd. The action is always dev→prd whichever row you launch it from (the pipeline list comes from the git remote, and the pipeline itself decides where it deploys), so the title says that outright instead of the menu's placement implying it — and the entry is no longer hidden on the connection you actually work in
• A queued deployment now says so. Starting a second pipeline while the first is still running leaves it waiting for an agent, and the toast reports Queued — waiting to start with its own clock until Azure DevOps says the run has actually begun. The progress bar no longer moves during that wait, so it no longer arrives at 95% before the deployment has started
• The bar is paced by run time, not queue-plus-run time — the calibration sample is now the last five succeeded builds (finishTime − startTime), which is the only place the real start of a run is visible. Historical queue waits used to inflate the median and slow the bar down
• The outcome toast splits the two: Pipeline 'x' succeeded in 4m 12s (queued 2m 3s), with the queue part shown only when it was longer than 30 seconds
• A run canceled while still in the queue, or an unreadable queue state, are both handled: the first is reported like any other outcome, the second falls back to watching the run rather than giving up on it
• Files touched: src/{devOpsDeploy,azureDevOpsClient,lakehouseProvider}.ts, package.json, README.md, CLAUDE.md

2.4.2

• Deploying moved to where it belongs: right-click a production connection → "Deploy…" and pick a pipeline from a quick list, sorted A–Z. Same run, same calibrated progress toast as before — only the way in has changed
• The DevOps Deployment Pipelines pane is gone, and with it the play button next to every pipeline. The list is short, flat and only ever wanted at the moment you deploy, so a permanently-visible pane was spending screen space on it all day; and putting it behind a production connection says what the action means, where a play icon on every row is one misclick from a production release
• The picker opens instantly, showing its own loading indicator while the pipeline list arrives, rather than leaving the click unanswered
• Nothing else about deployment changed: the run still goes to the pipeline's default branch, the bar is still paced by the median of the last five succeeded runs and holds at 95% until DevOps reports the real result, cancelling the toast still stops watching without cancelling the deployment, and every outcome toast still links to the run
• Files touched: src/devOpsDeploy.ts (was devOpsPipelinesProvider.ts), src/{extension,lakehouseProvider}.ts, package.json, README.md, CLAUDE.md

2.4.1

• New "DevOps Deployment Pipelines" section — deploy the solution without leaving VS Code. It lists your repo's own Azure DevOps pipelines, grouped by their DevOps folder, with a play button on each row that queues a run on its default branch. Organization and project are read from the repo's git remote, so there is nothing to configure
• The click answers straight away. The row's icon starts spinning the instant you press play (and shows running…), and the progress toast opens immediately with "Starting…" rather than after the two calls it takes to queue a run — so a deploy button never looks like it was ignored. The row keeps spinning after you dismiss the toast, so the tree shows what is deploying right now, and when the run ends it shows a ✓ or ✗ for a few seconds — worth having because a success toast hides itself
• The progress toast is paced by how long the pipeline actually takes. It takes the median of the last 5 succeeded runs and walks the bar to 95% over that time, then holds at 95% — with the elapsed clock still running — until DevOps reports the real result. A pipeline that has never succeeded gets an honest elapsed-time readout instead of a bar that would be guessing. Every outcome toast links straight to the run in the browser
• Cancelling that toast stops watching — it never cancels the deployment. The ✕ is one click with no confirmation, so it must not be able to kill a production release; the toast says the run continues and gives you the link
• Signing in takes no configuration. Pipelines use your own az login identity (not a connection's service principal), and the view asks DevOps which tenant the organization belongs to rather than guessing — so it works no matter which customer tenant az happens to be signed in to, without a re-login, as long as you have signed in to your own tenant at some point. If sign-in genuinely fails, the view says so and tells you what to run. There is a lakehouseStudio.devops.tenantId escape hatch for networks where that lookup cannot get through
• Fixed: notebooks no longer sit on "No kernel" while an engine is running. A notebook now attaches itself to the right kernel — the Spark session when one is up, the Local engine otherwise — so Run all stops opening the kernel picker. The main cause was that each Spark session gets a new kernel identity, so a notebook's remembered choice silently stopped resolving as soon as a session was restarted. Also fixed along the way: a notebook opened while a session was busy with a long run got no kernel at all until it finished, and a notebook left on the engine you just stopped now moves to the one you started. If you picked a specific subsession by hand, that choice is kept
• The Fabric Pipeline Explorer is gone. It had been hidden since 2.3.2 and unused since — Fabric data pipelines are run through the MCP server's run_pipeline, which prompts for parameters and reports activities the same way. Its code, commands and REST calls are removed; MCP run_pipeline is entirely independent and unaffected
• Files touched: src/{azureDevOpsClient,devOpsPipelinesProvider}.ts (new), src/{extension,fabpyNotebookController,aileronController,authProvider,fabricApiClient,setupMcpServer}.ts, src/{pipelineProvider,pipelineMonitorProvider}.ts (deleted), package.json, README.md, CLAUDE.md

2.4.0

• The MCP server now uses the engine you have running — it no longer starts one of its own. With the Local engine active, Claude's queries go into that kernel: the same connected engine, the same resident cache your own queries filled, the same temp views. Previously the server built a private aileron engine inside itself, which meant every Claude query paid a cold start against data your kernel already had in memory, and ran under Azure CLI auth even when the connection uses a service principal (the kernel's identity is now whatever the extension connected with)
• Nothing running means a refusal, not a silently different engine. Both backends are attach-only now, so "the engine you asked for is not running" has one honest answer: the server says so and names the two status-bar items that fix it. The Livy→local fallback introduced in 2.3.1 is gone — its premise was an in-process engine that could answer with VS Code closed, and attaching to a kernel VS Code owns removes that premise entirely
• A server pinned to a different lakehouse than the kernel refuses instead of answering. Each MCP server is pinned to one connection at startup while the kernel follows the extension's active connection, so they genuinely can disagree; answering anyway would read (or write) another customer's lakehouse. The message names the fix. Practical consequence: while the Local engine is up, the MCP server that answers is the one matching your active connection
• run_notebook runs in the kernel too. The python -m aileron.mcp.nbrun child process is gone; notebooks go through the notebookutils shim the extension already installs in the kernel, exactly as the Spark path uses the shim in the Livy session. Side effect worth having: the whole class of "the child loaded a stale extension's shim" bugs (the reason 2.3.1 added --extension pinning) cannot happen any more. dag_concurrency is now ignored on both engines and says so in the report — it was only ever honoured inside that child
• Result rows cross the wire as Arrow, not text. The server and the kernel run the same shared-venv interpreter, so results round-trip as an Arrow IPC buffer: decimals and timestamps arrive exactly as the engine produced them instead of being re-parsed from strings, and there is one less type mapper to drift
• Starting the Local engine from the status bar now connects it too. Previously the kernel connected to your lakehouse lazily, on the first query you ran — so a freshly-started kernel was green but not yet attachable, and the MCP server refused while telling you to do exactly what you had just done. Green now means attachable
• The Local engine kernel now auto-loads your utility/*.py modules, once per kernel, exactly as a Spark session has always done at start-up. This is what makes MCP run_notebook correct on the Local engine: the notebookutils shim strips a from <util> import * whose name is not a workspace notebook, assuming start-up auto-load already put those names in the namespace — true on Spark, false here, so a notebook lost both its helper functions and the imports they carry and died partway through on something like NameError: name 'StructType' is not defined. Running the same notebook by hand always worked, because that path inlines the modules per cell — which is exactly why the failure looked like an MCP bug
• Fixed: a failed notebook run reported a traceback with no error on it. The per-notebook error was capped at the first 300 characters, and a traceback's exception message is its last line — one long Windows temp path in the first frame was enough to bury the entire cause. It now shows the tail, so the thing that actually went wrong is the thing you read
• One kernel means one queue. Claude's query and your F5 are serialized against each other by ipykernel (first come, first served), the Local engine status-bar item spins with a - MCP query suffix while Claude works, and a statement that times out says whether it was still queued. Your Escape still interrupts the kernel, which will also stop a query Claude has in flight — the price of sharing one engine
• Files touched: src/{aileronKernelManager,aileronController,utils,extension,sparkSessionManager,setupMcpServer}.ts, python/mcp-server/ecit_lakehouse_mcp/{local_engine,engine_router,livy_engine,render,server,tools,selftest,__main__}.py, python/mcp-server/pyproject.toml, python/tests/{test_mcp_local_kernel_engine,test_mcp_engine_router,test_mcp_livy_engine}.py, package.json, README.md, CLAUDE.md

2.3.3

• The third status-bar item is gone. The read-only derived-engine indicator (it appeared on .sparksql tabs and said which engine F5 would use) restated what the two engine items already show — the engine is whichever one is green — so the same answer sat on the row twice, and being read-only it was the one thing there you could not click
• Fixed: notebook Play in the tree refused to run on the Local engine. The play button's precheck asked "is a Livy session up?", which is Livy-only, so with the Local engine running and Spark stopped it showed "No active Spark session" and refused a notebook the queue runner would have executed fine (that runner has been engine-aware since v2.1.0). The precheck is gone; the queue manager, which knows about both engines, owns the decision. Play now also lazy-starts the Local engine when nothing is running, exactly like F5 — previously it demanded you start an engine by hand first
• Fixed: deploy_semantic refused every repo whose parameter.yml carries a dev: key — that is to say, every real customer repo, since the release pipeline needs that key to exist. The rule was inverted: it treated substitution as the danger, when fabric-cicd's substitution fails open, so the genuinely dangerous file is the one with no key for our environment — it publishes the model unrewritten and logs a warning identical to a successful rewrite. A dev: key pointing at this connection's own lakehouse is a no-op rewrite and keeps the local deploy on the same code path the release pipeline uses; a parameter.yml without one is now what gets refused, naming the fix. No parameter.yml at all is fine (nothing to substitute)
• Fixed: a dev: key also blocked refresh_semantic, which publishes nothing and runs no substitution whatsoever. The parameter.yml check is now publish-only
• The MCP tenant pre-flight stopped crying wolf. It compared az's active tenant against the connection's and complained on any difference — but one machine has one az context, so a consultant with three customers open was "wrong" for two of them at all times, while every token path here and in the extension is tenant-pinned and works regardless (az account get-access-token --tenant <id> mints for any tenant already in the CLI cache). It now probes the connection's own tenant the same way the real token path mints, and only mentions the active tenant when that probe actually fails, as the likely cause. This was advisory noise at server startup, but a hard failure inside deploy_semantic — a deploy died on it while signed in elsewhere, despite the deploy itself being fine
• Files touched: src/{extension,notebookQueueManager,aileronKernelConfig}.ts, python/mcp-server/ecit_lakehouse_mcp/{connections,gate,tools,server,smrun,selftest}.py, package.json, README.md, CLAUDE.md

2.3.2

• Results pane renamed from "Lakehouse Results" to "Lakehouse Data Viewer"
• MCP query now renders 10 rows by default instead of 100. The footer still reports the true total, so a small default never hides how much matched — but a hundred wide rows per exploratory query was crowding Claude's context with data it never read. Raise max_rows deliberately when you need the rows, and the tool now tells Claude to project the columns it needs rather than SELECT * when it does
• MCP recovers from a wrong az login without a restart. Fixing the login and retrying already worked when the first token failed to mint; three things had outlived the mistake and no longer do: a successfully-minted wrong-tenant token was cached for up to an hour with no invalidation (now dropped on any 401/403, so the retry reaches a fresh identity), the startup tenant-check warning was frozen for the server's lifetime and kept warning after the login was fixed (now re-taken on demand behind a 60s TTL, and it goes quiet once the tenant matches), and the Local engine instance was memoized forever, sealing in whatever identity built it (now rebuilt after an auth-shaped failure — plan errors still reuse it, no cold-start tax). Note the MCP server only ever uses Azure CLI auth: it cannot read VS Code SecretStorage, so a service-principal connection is not an alternative here
• Pipeline Explorer is hidden. The view and its Run/Cancel commands are unused — pipelines are run through the MCP server's run_pipeline. Deactivated rather than deleted, so it is a one-line revert
• Fixed: .pytest_cache/ was shipping inside the VSIX. pytest writes its own .gitignore in that folder so git never flags it, but vsce has no such courtesy — and the release runs pytest two gates before vsce package, so every release carried it
• Files touched: package.json, .vscodeignore, python/mcp-server/ecit_lakehouse_mcp/{server,tools,render,fabric_jobs,livy_engine,local_engine}.py, python/tests/test_mcp_livy_engine.py, README.md, CLAUDE.md

2.3.1

• Fixed: MCP run_notebook on the Local engine died with ModuleNotFoundError: notebookutils before any cell ran. The notebook child process picked its notebookutils_shim.py by globbing the newest installed extension, which served a stale pre-2.3.0 shim whenever the running extension was newer (or an F5 dev host). active_engine.json now carries the running extension's own path and the child is pinned to it
• MCP falls back Livy→local when Spark is active but no session exists to attach to — the common case being VS Code closed while a Spark session was running, leaving a stale active_engine.json. The server re-derives the same rule the extension would, answers from the Local engine, and says so in a one-shot note on the result. It never falls back on a session that is still starting (you are choosing Spark), on a session that died mid-statement (partial side effects), or when the engine wheel is absent — that still refuses with "start it in VS Code"
• Fixed: a killed Livy session surfaced a raw Livy 400 instead of self-healing. Flipping engines kills the session, and a statement POST to a killed session answers 400, not 404 — the recovery path only handled 404. Submitting now re-discovers the session once on 400/404/410 and retries (nothing had run, so the retry is safe); auth errors still surface raw rather than being masked as "no session". The session list also gets one grace re-check, since it can briefly lag a just-started session
• The Local engine status-bar item now spins during MCP work — both Claude's local queries and local run_notebook runs — instead of leaving the Spark item spinning for work Spark never did. Both items show a - MCP query suffix when the activity is Claude's rather than yours, and a crashed server can no longer leave either one spinning
• Files touched: python/mcp-server/ecit_lakehouse_mcp/{engine_router,livy_engine,local_engine,server,tools,selftest}.py, src/{extension,sparkSessionManager,utils}.ts, python/tests/test_mcp_engine_router.py, package.json, README.md, CLAUDE.md (+ a Releasing section documenting the release gates, and .claude/skills is now tracked in the repo)

2.3.0

• The Local engine is back, as a first-class citizen. The v2.2.0 removal is reversed: the extension now has two engines — Fabric Spark (Livy) and the Local engine — with at most ONE active at a time. The active engine is derived from which one is running (no per-tab choice, no toggle): starting the local kernel ends the Livy session and vice versa, and F5, table previews, Ctrl+3, and notebook affinity all follow. A read-only status-bar indicator shows which engine F5 will use
• The T-SQL SQL-endpoint query path is retired completely — the per-tab toggle, the tedious client, the SQL Endpoint submenu, Refresh SQL Endpoint metadata, and the connection's SQL-endpoint fields are all gone. The Local engine covers the fast-read case, Spark covers everything
• One MCP server, two backends: the bundled ecit_lakehouse_mcp server now routes every call to whichever engine is active in the extension (via local_development/active_engine.json, written on every engine transition, read per call — switching engines retargets a running server with no restart). Same tools, same pseudonymisation, same prd read-only gates on both backends. run_notebook runs on the Livy session (Spark active) or in a disposable local child process (Local engine active). When the file says local but the engine is not installed, tools refuse with guidance naming Setup local engine — never a silent engine switch. An "unsupported by the local engine" error explicitly suggests switching to Spark rather than concluding the query is wrong
• Engine install UX: a one-time per-machine activation offer ("Install the local engine?") plus the dimmed status-bar item's Install modal; Setup local engine installs the wheel + platform baseline into the SAME shared venv as the MCP server and runs the MCP selftest as a canary afterwards. Uninstall local engine is now pip uninstall only — the shared environment is kept for the MCP server
• Silent engine updates at activation are back (wheel feed latest.json, SHA-256 verified), as are the lakehouseStudio.aileron.wheelFeed/wheelTenantId/wheelSource settings (aileron.pythonPath stays retired — mcp.pythonPath is the one shared interpreter key)
• Files touched: src/sqlEndpointExecutor.ts deleted (+tedious dependency, −21 packages); 8 engine files restored (aileronKernelManager, aileronController, aileronKernelConfig, kernelTransport, jupyterWsClient, localKernelServer, engineFeed, setupLocalSpark) + python/kernel-requirements.txt; python/mcp-server/ecit_lakehouse_mcp/{local_engine,engine_router}.py new; extension.ts, dataFrameViewerProvider.ts, utils.ts, venv.ts, setupMcpServer.ts, mcpJson.ts, shim, tests, package.json, docs

2.2.1

• Self-migration from the local engine: opening a repo with connections but no current MCP entry shows a one-time "Set up the ecit-lakehouse MCP server?" toast — Install runs Setup, which first detects and removes a leftover pre-2.2.0 engine environment (stops everything running from it, deletes the multi-GB venv, clears the dead lakehouseStudio.aileron.pythonPath setting), then installs the MCP server fresh. No manual uninstall needed on machines that had the engine; "Not now" is remembered per repo
• Setup converges all MCP entry generations: re-running Setup (or clicking Install) now also replaces the old hand-written lhEwh-fabric-* / ms-fabric-mcp entries in .mcp.json, not just the previous aileron.mcp ones — matched by command basename, since those entries carry no args to match on
• pip no longer builds inside the extension folder: the bundled MCP package is staged to a temp directory before installing, so the live extension install stays clean and locked-down or synced profiles don't break Setup
• Results pane renamed from "DataFrame Results" to "Lakehouse Results", with a friendlier empty-state text
• Key Vault Name field is now authoritative — it is injected directly as KEY_VAULT_NAME into Livy sessions (previously only the generic env-var rows were injected, so editing the field on an existing connection silently did nothing). The key/value rows are removed from the connection dialog — across all real connections they only ever duplicated the Key Vault name; environmentVariables in settings.json still injects if hand-edited. A one-time migration moves the env-var value back into the field
• Fixed: switching connection mid-session left KEY_VAULT_NAME pointing at the previous connection's vault — the retarget now re-injects it (or removes it when the target connection has none)
• Files touched: src/setupMcpServer.ts, src/mcpJson.ts, src/venv.ts, src/extension.ts, src/connectionManager.ts, src/sparkSessionManager.ts, src/fabpyNotebookController.ts, src/dataFrameViewerProvider.ts, src/utils.ts, src/models.ts, package.json, README.md, CLAUDE.md

2.2.0

• The local engine is removed. Spark via the Livy API is the single execution engine for notebooks, F5, previews, table operations, and the tree runner; the read-only T-SQL endpoint remains for .sparksql tabs. The engine, its Setup/Uninstall/start-stop commands, the wheel-feed settings (lakehouseStudio.aileron.*), and its status-bar item are all gone
• The MCP server now runs on the extension's Livy session — the bundled ecit_lakehouse_mcp package (pure Python, ships in the VSIX) attaches to the already-running session, never creates one, and refuses with "start it in VS Code" guidance when none is running. Same tools and pseudonymisation as before; Setup ecit-lakehouse MCP server needs nothing but Python on the machine
• T-SQL endpoint is always available again: the lakehouseStudio.showTsqlEndpoint test setting is removed. The status-bar item toggles the focused .sparksql tab between Spark (Livy API) and the T-SQL endpoint; every tab defaults to Spark
• Setting renamed: lakehouseStudio.aileron.pythonPath → lakehouseStudio.mcp.pythonPath (the shared venv now exists for the MCP server; re-running Setup writes the new key)
• MCP setup/update hardening: orphaned venv processes are swept before every pip install, so a hard-killed extension host can no longer make the next install fail on locked files
• Files touched: src/venv.ts (new), src/extension.ts, src/setupMcpServer.ts, src/mcpJson.ts, src/dataFrameViewerProvider.ts, src/notebookQueueManager.ts, src/notebookSequenceRunner.ts, src/fabpyNotebookController.ts, src/sparkSql*, src/sqlParser.ts, src/queryHistory.ts, python/notebookutils_shim.py, 8 files + python/kernel-requirements.txt deleted, package.json, README.md, CLAUDE.md

2.1.20

• Fixed: the "CLAUDE.md changed" toast fired on every branch switch — reported as ~6 times a day. The dedupe stored only the last notified hash, so hopping between two branches whose CLAUDE.md differ flips the file A→B→A→B and every checkout mismatched that one hash. It now remembers the last 50 versions it has seen (claudeMd.seenHashes), so a version toasts once, ever, and switching back to an already-seen version is silent
• Two related cases that also toasted needlessly: a first-ever run now records the current file silently (a fresh clone's CLAUDE.md is a baseline, not a change), and your own editor save records its hash immediately instead of only being time-suppressed — previously your own edit could still toast at the next activation
• Removed the duplicate "Sign in to Azure/Fabric..." right-click entry. Azure CLI connections showed two login commands side by side; the surviving Login to Azure... pins az login --tenant <connection tenantId> and is strictly better than the plain az login it replaces. The auth-error toast's Login... button now runs the tenant-pinned one too, so a 401 no longer offers to sign you in unpinned
• The login progress toast now sets expectations — that the browser sign-in window may open behind other windows, and that completion takes ~20 s after you sign in. The delay is az login enumerating subscriptions before it returns, so it is the CLI's, not ours; the connection dialog's own status text says the same
• Files touched: src/claudeMdWatcher.ts, src/extension.ts, src/utils.ts, src/connectionManager.ts, package.json, README.md, CLAUDE.md

2.1.19

• .mcp.json entries now carry env: { "AILERON_MCP_REDACT": "1" }. "1" is the MCP server's own default — results are pseudonymised (engine D-174) — so this changes no behaviour; it exists so the off-switch is discoverable. Without it there is nothing in the repo to suggest the control exists, and a colleague who has decided a lakehouse holds no personal data has no way to find that out except by reading the engine's source. Flip the value to "0" and reload the window to disclose values for a session. The narrower alternatives are unchanged: revealColumns on the connection discloses named columns permanently, AILERON_MCP_REVEAL does it for one session
• Files touched: src/setupMcpServer.ts, package.json, README.md, CLAUDE.md

2.1.18

• Auto-load module failures are now loud. Every utility/*.py module loaded at Spark session start gets a line in the LakehouseStudio: Livy (Spark session) channel, and a failure raises a warning toast naming the modules with a Show log button. They were written to the dev-tools console only, i.e. invisible. This matters because customers moved their Spark conf (caseSensitive, ANSI off, the parquet rebase modes) out of the Fabric environment and into dataplatform_functions.py — where an environment was guaranteed by Fabric, a module load is best-effort, and a module that silently doesn't load gives you a session that computes different answers rather than one that errors
• A deleted Fabric environment no longer breaks session start. The environment picked at session start is saved on the connection and then read blind by every other start path (F5, notebook cells, expired-session recreate), so deleting a workspace environment left each connection requesting a GUID that no longer exists — with a Fabric error naming neither the environment nor the setting holding it, across every connection on the machine. The saved environment is now verified before use (one extra call, and only when one is saved) and cleared when it's gone, falling back to the starter pool. An unreadable environment list keeps the saved value rather than dropping it, so a transient error can't silently swap a working environment for the starter pool
• Note on Fabric environments generally: the environment attached at session start is decided by how many environments the workspace has — zero attaches the starter pool, exactly one is auto-attached with no prompt, two or more show a picker. The workspace's default environment setting is not consulted, and "starter pool" is not offered as a picker entry, so a workspace holding one environment always attaches it
• Files touched: src/sparkSessionManager.ts, src/extension.ts, package.json, README.md, CLAUDE.md

2.1.17

• New connections default to Azure CLI / Interactive instead of Service Principal (SP is still fully supported and selectable). A service principal exists for unattended execution — this extension is attended by definition, and deployed pipelines run under Fabric's own identity, not these connections. Meanwhile a shared SP erases who did what in Delta commitInfo and the Fabric monitor, and asserts "the platform process did this" when a consultant actually ran it by hand
• Interactive credentials are now scoped to the connection's tenant. They previously shared one unscoped credential, which resolved against whichever tenant az happened to be on — survivable when SP was the default, a correctness bug now that CLI is. Requires no extra logins: az mints tokens for any tenant already in its cache (verified across four customer tenants while signed in to a fifth)
• connection.tenantId is now load-bearing, not cosmetic — it decides which tenant the token comes from. Set it on every connection
• Fixed: saving a connection as Azure CLI wiped its Tenant ID. Correct while the field was Service-Principal-only, silently destructive once it began scoping the CLI credential — the connection fell back to whatever tenant az was on, and the wrong-tenant warning went blind. clientId is still cleared, since that genuinely is SP-only
• Unused service principal credentials are pruned at activation — any connection no longer using a service principal has its stored client secret and stale clientId removed, so an unusable secret can't linger on disk. Note this means switching a connection back to Service Principal requires re-entering the secret
• Tokens are now cached per (identity, scope) with a 5-min expiry margin. The CLI credentials have no in-process cache and re-spawn az on every token request — measured ~710 ms per call — which would have taxed every tree expand, preview, and capacity check under the CLI default. Concurrent requests share one in-flight fetch; SDK clients (OneLake/ABFSS) go through the same cache
• An SP connection with incomplete credentials now fails loud instead of silently running as Azure CLI — a degraded connection must not be indistinguishable from a deliberate one, and the secret pruning above makes the missing-secret case reachable
• Note for CLI connections: Capacity Resume/Pause/Scale needs Azure RBAC (Microsoft.Fabric/capacities/*) on the user, not just a Fabric workspace role
• Files touched: src/authProvider.ts, src/connectionManager.ts, src/fabricApiClient.ts, src/extension.ts, package.json, README.md, CLAUDE.md

2.1.16

• Workspace Git sync — the connection row now shows ⇧ N to sync in yellow when the Fabric workspace is behind Git, and right-click → Update workspace from Git applies the changes with Git winning every conflict (the "Update all" nobody remembered to run). No confirmation prompt: reaching the command is already an explicit act, and re-syncing is how a sync is undone
• Git calls use your Azure CLI identity, never the connection's service principal. Fabric's Azure DevOps integration acts as the calling identity and each identity needs its own Git credentials registered in Fabric — an SP has none, so every SP call fails GitCredentialsNotConfigured regardless of workspace permissions. Syncing is a human action anyway, so the workspace recording the developer is the truthful attribution
• Pending updates are only ever Git → workspace. changes[] mixes both directions; workspace-only entries are the phantom diffs we handle elsewhere, and counting them would put a permanent nonzero badge on every connection. Note workspaceHead != remoteCommitHash is not the signal — a workspace routinely trails the remote with nothing to sync
• Checked on activation, connection switch, Explorer visibility, a .git refs change, switching to main/master (15s later), and a periodic timer — lakehouseStudio.workspaceGitCheckInterval, default 10 min, 0 disables. The timer is the load-bearing one: a PR merged in the Azure DevOps web UI changes no local ref, so no git-based trigger can fire. Also Lakehouse Studio: Check Git status for workspace for a deterministic check
• Dev only — production connections are excluded entirely (no badge, no command, no API call); prd goes through the deployment pipeline. Targets notebookWorkspaceId explicitly with no fallback, since falling back to the lakehouse workspace would offer to overwrite the wrong workspace's items
• New output channel LakehouseStudio: Workspace Git Sync logging every decision — the feature fails soft so a broken API can't break the tree, which made "no badge" indistinguishable from "up to date"
• Wrong-tenant warning: when the Azure CLI is signed in to a different tenant than the active connection names, a toast offers Sign in (runs az login --tenant <connection.tenantId>). Fires for every connection with a tenant ID — even on an SP connection the CLI context is what semantic-model deploys, the MCP server and your own terminal work pick up
• Tenant ID is now shown for both auth methods in the connection dialog (was Service-Principal-only) — it has always pinned az login --tenant regardless of auth method, and it is what the warning above compares against. Switching auth to Interactive no longer clears it
• CLAUDE.md change notification: a dismissible toast when CLAUDE.md changes (typically a new version arriving from a branch switch or pull), since it only takes effect in a new Claude Code conversation. Off switch: lakehouseStudio.notifyClaudeMdChanges
• Connection row trimmed: workspace and lakehouse names removed — measured across 21 real connections they discriminate nothing (19 lakehouses named lhEwh; 13 connections sharing two workspace names), and they pushed the dynamic, actionable segments rightward. Both remain in the tooltip
• Files touched: src/workspaceGitSync.ts (new), src/claudeMdWatcher.ts (new), src/fabricApiClient.ts, src/authProvider.ts, src/connectionManager.ts, src/lakehouseProvider.ts, src/fabricDecorationProvider.ts, src/extension.ts, package.json, README.md, CLAUDE.md

2.1.15

• MCP server entries are now named ecit-lakehouse-<env> (was lakehouse-<env>) — matching the command's own name, so our servers are visibly ours in an MCP client's list next to the customer's own entries. Re-run Lakehouse Studio: Setup ecit-lakehouse MCP server in each repo to pick up the new names
• No migration step, by construction: entries are owned by their args shape (-m aileron.mcp), never by key, so re-running setup strips the old-named entries and writes the new ones — and the uninstall still finds entries written under either name. The only residue is a now-inert mcp__lakehouse-<env>__* allow rule in .claude/settings.json, left alone deliberately (we only ever add rules, so we never risk deleting one a user added by hand)
• Files touched: src/setupMcpServer.ts, src/mcpJson.ts, package.json, README.md, CLAUDE.md

2.1.14

• The engine auto-update now clears its own blockers — it stops a local kernel that warmed up in this window (checked after the wheel download, where a restored notebook's warm-up races it), sweeps venv processes orphaned by hard-killed extension hosts, and retries pip over Windows' handle-release lag. This fixes the startup "the previous one is still in place" failure that a later manual Setup made look intermittent
• Orphaned-process sweep at activation: leftover jupyter-server → python.exe chains from hard-killed hosts (F5 dev-host stop, crash) are found by their dead parent PID and taskkilled — a live kernel or MCP server in another window has a living parent and is never touched. Closes the long-documented open item (eight orphaned chains measured on one laptop)
• Engine-update failure toast gets a Show output button and the setup log now brackets each update with clear header/succeeded/FAILED lines, distinguishing a pip failure from a verify-import failure
• New setting lakehouseStudio.showTsqlEndpoint (default true): turn off to hide the whole T-SQL endpoint surface — the status-bar toggle, the table right-click "SQL Endpoint" submenu, and "Refresh SQL Endpoint metadata" — while testing the local engine as the only query path. Flipping it back restores everything, no reload needed
• Semantic Model Workspace nag: while any project connection is missing its Semantic Model Workspace ID, activation shows a warning naming the connections with an "Edit connection" shortcut — so all connections get the field the MCP server's semantic-model tools need (temporary measure)
• Schema compare removed — unused; the extension_state.json persistence it was the only consumer of went with it. Recoverable from this release's commit if ever needed
• Files touched: src/schemaComparisonProvider.ts (deleted), src/extension.ts, src/setupLocalSpark.ts, src/localKernelServer.ts, src/utils.ts, src/models.ts, package.json, README.md, CLAUDE.md

2.1.13

• MCP setup now pre-approves the servers in .claude/settings.json — one mcp__<server>__* rule per entry in permissions.allow. Registering a server made it available but not frictionless: Claude Code asks to approve every MCP tool it has no rule for, which in plan mode is a prompt per query
• The rules cover every server in .mcp.json, not just the ones we wrote — including the previous-generation lhEwh-fabric-* servers still in the customer repos during the overlap. Half-silenced prompts are still prompts
• Additive and idempotent: an existing permissions.allow is appended to, a permissions block without allow is filled in, the file and .claude/ are created when absent, and every other key (hooks, env, deny, …) rides through untouched. Re-running adds nothing
• A settings.json we cannot safely merge into is never rewritten — unparseable JSON, or a permissions.allow that is not an array, warns and leaves the file alone. The MCP registration itself still succeeds
• Written to settings.json, not settings.local.json: the servers are a property of the repo, so the rules belong in the file the team shares. A rule naming a server the reader does not have is inert
• Uninstall deliberately leaves the rules behind — an allow rule with no server behind it matches nothing, while removing rules risks deleting one added by hand
• Files touched: src/claudeSettings.ts (new), src/setupMcpServer.ts, package.json, README.md, CLAUDE.md

2.1.12

• MCP setup now installs the semantic-model deploy packages (fabric-cicd==1.1.0, azure-identity>=1.17) — what the engine's deploy_semantic / refresh_semantic tools need to publish a .SemanticModel folder to Fabric. Only attempted on an engine that actually has those tools (0.1.18+), so an older environment is left exactly as it was
• Installed by the MCP command, not by Setup local engine. These are MCP-only: a notebook kernel never deploys a semantic model, so they stay out of the kernel baseline and ride the same path the mcp SDK already does
• A failed install warns and continues rather than aborting setup — the server still serves its other seven tools, and the two deploy tools refuse with the exact pip line for that interpreter instead of breaking anything. The full pip message is in the "LakehouseStudio: Setup ecit-lakehouse MCP server" output channel
• fabric-cicd is pinned, not floored, on purpose: its parameter substitution is an exact case-sensitive key match that fails open, so which version runs decides whether a model pointing at the wrong lakehouse can be published silently. Keep the pin equal to the engine's own deploy extra
• Files touched: src/setupMcpServer.ts, package.json, README.md, CLAUDE.md

2.1.11

• Semantic Model Workspace ID on the connection — a new optional field in the Add/Edit Connection dialog's Workspaces section, with the same Select... workspace picker as the Lakehouse and Notebook workspace IDs. It records where this lakehouse's semantic models live, which is what lets the ecit-lakehouse MCP server walk the chain from a gold table to the model that consumes it
• Nothing was added to .mcp.json for it. The MCP server already resolves --connection <id> by reading the connection record out of VS Code's global settings.json, so the GUID reaches it the moment it is saved — a second channel for a datum the server can already see would only be a second thing to keep in sync
• No fallback to the lakehouse workspace, deliberately the opposite of Notebook Workspace ID. A semantic-model workspace is a different workspace often enough that defaulting would be wrong more often than right, and a wrong GUID here is worse than an absent one: empty means "not configured"
• The Add Connection dialog now opens on Service Principal — it is what new connections actually use. Editing an existing connection is unaffected: a record saved before the auth field existed still reads as Interactive, so nothing gets silently re-labelled. Note that clicking Select... before filling Tenant/Client/Secret now fails with the usual auth error — switch the dropdown to Interactive if you want to browse straight after az login
• Files touched: src/connectionManager.ts, src/models.ts, package.json, README.md, CLAUDE.md

2.1.10

• Generate script... → OPTIMIZE — a fourth entry on the table right-click menu, in its own section below the write templates. Opens a .sparksql tab with OPTIMIZE <schema>.<table> pre-selected (so F5 runs just it), plus commented VORDER and VACUUM variants ready to uncomment. V-ORDER is the reason it exists: it is what a gold table feeding a semantic model wants, and it had been folklore rather than a template
• Fixed: VACUUM was silently dropped on the Spark engine. It matched neither statement bucket in the executor, so it fell through and reported "0 statements executed successfully" — a green light on a statement that never ran. Anyone uncommenting the new script's VACUUM line would have hit it
• The local engine now updates itself. The update check runs on every activation (when the engine is installed) and installs a newer version silently, ending in a single toast: Engine has been upgraded to vX.Y.Z, restart kernel if it is active. It previously asked, behind a 24-hour throttle — and the two combined were the problem: the timestamp was written before the prompt, so dismissing once cost the whole day, and you could sit several versions behind for a week without seeing a second prompt. On the one channel that carries an engine bug fix to you. A progress notification still shows while an install runs
• The update toast has no button. It used to offer Restart kernel, which runs the kernel toggle — a command that starts a stopped kernel. At activation the kernel is always stopped, so the button reliably cold-started an engine nobody asked for (this supersedes the 2.1.9 note that the daily toast keeps it)
• The "engine too old" warning no longer fires for a problem we just fixed — it is now evaluated after the upgrade, against the resulting version. Offline behaviour is unchanged: an unreachable feed still warns against what is installed
• Update failures now name the cause you can act on — a running local kernel holds the engine files open on Windows, so pip cannot replace them. Stop the kernel (in every VS Code window) and reload
• Verified, no change needed: re-running Setup ecit-lakehouse MCP server does not touch hand-written .mcp.json entries such as lhEwh-fabric-dev / lhEwh-fabric-prd. Ownership is decided by the entry's args shape (-m aileron.mcp), never by its key, and the write merges into the parsed file rather than rebuilding it
• Files touched: src/extension.ts, src/setupLocalSpark.ts, package.json, README.md, CLAUDE.md

2.1.9

• Lakehouse Studio: Uninstall local engine (palette): the inverse of Setup, so install → test → reset is a loop rather than a one-way door. Stops the kernel, deletes the Python environment, clears lakehouseStudio.aileron.pythonPath, forgets the daily update-check timestamp, sweeps the leftover %TEMP%\aileron-wheel-* download folders (nothing has ever cleaned those up — ~50 MB per install), and strips this repo's MCP entries, which can only point at the interpreter being deleted. One modal confirm naming every path and key
• Lakehouse Studio: Uninstall ecit-lakehouse MCP server (palette): removes this repo's aileron.mcp entries from .mcp.json, deleting the file outright when we were its only content. Entries are matched by shape (args: ['-m','aileron.mcp', …]), never by key name — so a customer's own lakehouse-* entry is safe and a renamed key of ours is still found. Deliberately does not pip uninstall mcp: the SDK is part of the engine baseline, so it goes with the environment, not with the registration
• A failed environment delete no longer leaves a half-uninstalled state. rmSync walks depth-first, so by the time it hits a locked file it has already removed a great deal — usually including the engine itself. Aborting there produced the worst possible outcome: import aileron fails so the status bar goes grey and looks uninstalled, while the interpreter setting still pointed at the wreckage. The cleanup now always completes, and a partial delete ends in a warning naming the leftover folder with a Show folder button instead of a success toast
• Fixed a silent process leak in the local kernel server. proc.kill() signals only the direct child, but python -m jupyter server re-execs through a console-script shim, so one server is a chain of three processes. The tail survived every stop, and a live python.exe is memory-mapped — it holds the whole environment open. Eight orphaned server chains were found on one laptop, all parented to extension hosts that had already exited, each holding a port and its share of RAM. Windows now gets taskkill /T /F
• MCP setup no longer asks which connections to expose — it registers the ones belonging to the open repo. .mcp.json is per-repo, so this was never a real choice, and the picker was showing every connection on the machine across every customer (21 on one laptop, 2 of them the repo's) because the connection list is a global setting. The repo's own set is the folder-based filter the Lakehouse Explorer tree already uses
• Re-running MCP setup now makes .mcp.json match the repo instead of accumulating. A deleted or renamed connection used to leave its entry behind forever, pointing at an id that no longer resolves. Our entries are replaced; everything else in the file is preserved, and a derived name can no longer overwrite a key we do not own
• Both Setup commands now end with a Reload Window button — a reload restarts the extension host and the kernel with it, and Claude Code only picks up a changed .mcp.json on reload. This replaces the local-engine setup's old Restart kernel, which it strictly supersedes. The daily engine-update toast keeps Restart kernel, where a full reload would be overkill
• The dimmed "Local engine" status-bar item asks before installing. A single click used to start a multi-minute download and install with no warning — and the person clicking an item they have never used is exactly the one who does not know what it is. The prompt explains what the local engine does, that it takes about 5 minutes, and that it can be removed again
• Renamed the setup progress toast from "Setting up local Spark…" to "Setting up local engine…", resolved from the shared label helper rather than a literal — it is the most-seen string in the whole setup flow and had been saying Spark since the kernel stopped being Spark
• Files touched: src/mcpJson.ts (new), src/setupLocalSpark.ts, src/setupMcpServer.ts, src/localKernelServer.ts, src/aileronKernelManager.ts, src/aileronKernelConfig.ts, src/extension.ts, package.json, README.md, CLAUDE.md

2.1.8

• F5 now runs on the active engine, everywhere — the rule is simply "whatever engine is active", regardless of what the selection sits inside. Quick SQL Preview was the only execution entry point in the extension with no engine routing at all: it always submitted spark.sql(...) with no session id, which per Livy's contract auto-creates a session. So F5 on a selection inside a notebook cold-started Livy even when the local engine was the notebook's kernel — and under the single-active-engine rule that start then killed the local kernel out from under the notebook you were working in
• Fixed the engine binding leaking into the wrong key: recomputeEngineBinding derives a global fact ("which engine is active now") but wrote it per-tab whenever global engine mode was off, keyed by whatever editor happened to be focused — a vscode-notebook-cell: URI when you were in a notebook — leaving the real default stale, so tabs kept resolving to the previous engine
• "Delete schema" is finally reachable. The command has existed for a long time, but its menu was gated on a contextValue only set after a schema had been expanded and found empty, so in practice it appeared on nothing. It now shows on every schema like "Delete table" does, and since it is one click on a live schema the confirm names the blast radius (the table count) and carries the same PRODUCTION warning as "Restore table to version…". Still Spark/Livy-only — DROP SCHEMA is not in the local engine's supported set
• The command palette is curated. Ctrl+Shift+P offered all 72 commands, most of them tree right-click handlers that threw or silently did nothing when invoked without a tree item, plus two dead spinner placeholders and a pile of bare Delete / Rename / New folder entries. 59 are now hidden; the 12 that remain are the ones that actually work standalone, all under the "Lakehouse Studio" category. Gone from the palette: Local engine: start / stop / restart and Toggle T-SQL endpoint for SQL tabs, which only duplicated their status-bar click
• Renamed Setup local Spark → Setup local engine (it builds a venv and the local wheel — there is no Spark in it) and Setup MCP server → Setup ecit-lakehouse MCP server. The setup title is now a single shared constant: it was used as a notification button label and then compared back in two places, where a half-missed rename leaves a button that silently does nothing. Command ids are unchanged
• Files touched: src/extension.ts, src/aileronKernelConfig.ts, src/lakehouseProvider.ts, src/setupLocalSpark.ts, src/setupMcpServer.ts, src/localKernelServer.ts, src/engineFeed.ts, python/kernel-requirements.txt, package.json, README.md, CLAUDE.md

2.1.7

• Lakehouse Studio: Setup ecit-lakehouse MCP server (palette): registers the local engine's MCP server (python -m aileron.mcp) in this repo's .mcp.json, one entry per chosen connection — so MCP clients like Claude Code can query the lakehouse through the same engine environment the extension already uses. Palette-only like Setup local engine, but per-repo where that one is per-machine
• Connection-pinned, prd-safe entries: each entry is pinned by connection id (--connection <id>), resolved once at server startup — switching the extension's active connection never retargets a running MCP server. A connection tagged prd runs the whole server read-only (enforced server-side); the entry's LAKEHOUSE_MCP_ALLOW env var names the tools it may still run (execute, run_notebook, run_pipeline, or *) — Setup defaults a prd entry to the two run tools and a dev entry to *, semantic deploys are outside that vocabulary so nothing unlocks them, and Setup re-runs reset a hand-edit to the default. Entry names derive from the environment tag (lakehouse-dev, lakehouse-prd), falling back to the connection name
• Safe by construction: verifies the engine env exists and is new enough for aileron.mcp (≥ 0.1.14, offering Setup local engine when not), installs the mcp SDK into older envs, and proves the wiring with the server's own --selftest before anything is written. .mcp.json merging only touches the keys it derives — a hand-edited file that fails to parse is an error, never clobbered
• Engine baseline: MIN_ENGINE_VERSION raised to 0.1.14 (first wheel shipping aileron.mcp), so the daily update check pulls colleagues onto it; the mcp SDK joins the kernel-requirements baseline for fresh environments
• Files touched: src/setupMcpServer.ts (new), src/extension.ts, src/setupLocalSpark.ts, src/engineFeed.ts, python/kernel-requirements.txt, package.json, README.md, CLAUDE.md

2.1.6

• %%sql cells inside notebook.run / runMultiple ran as nothing at all — the most serious bug in this release. In the git format a magic cell is stored with every line commented (# MAGIC INSERT OVERWRITE …), and the Python shim never stripped that prefix: the cell reached exec as a block of comments, so it was a no-op with no output and no error, and the notebook then reported success. A silver build silently stopped happening while the DAG stayed green. Only child notebooks were affected — the interactive Run path has always stripped the prefix. An unsupported cell magic is now named in the error instead of dying with a SyntaxError pointing at generated code
• A notebook that reports its own failure is no longer counted as ok: the platform's exit_notebook() exits with {"status": "Failed", …} deliberately, to report a failure while letting the DAG continue. Neither Fabric nor we interpreted that, so such a notebook showed green in the progress grid even though it wrote nothing — observed on SkagenSSB, where five gold dimensions reported ok and produced no table. Only the status is reinterpreted: the activity stays out of errors, so dependents still run exactly as continue_on_error intends
• runMultiple returns Fabric's result shape: {name: {"exitVal", "exception"}} for every activity — succeeded, failed and dependency-skipped. It previously returned the bare exit value and dropped failures from the dict entirely, so the platform's own process_dag_results threw on every entry: a customer's DAG summary printed "parse error" for each notebook including the ones that succeeded, while the ones that actually failed never appeared at all
• runMultiple no longer deadlocks on a non-topological activity order: each worker used to wait on its predecessors' events while holding a pool slot, so if the first concurrency activities in list order all had unmet dependencies, no worker was left to run those dependencies and the DAG hung forever at zero CPU (Nordlux, 2026-07-29). Execution is now scheduled in topological waves (Kahn), so a bounded pool can never starve, the customer's list order survives wherever dependencies permit, and a dependency cycle fails loud naming its members instead of hanging
• Notebooks whose folder name and display name have drifted apart now resolve: a rename in the Fabric UI leaves the old folder name in git (or vice versa) — 9 real cases across 6 customers. The dynamic-DAG preload registers both spellings, since the shim's registry is an exact dict lookup with no fuzzy fallback
• One engine at a time: starting the local kernel now ends the Livy session, and a Livy session appearing stops the local kernel — silently, both directions, enforced at the kernel status hooks so every start path is covered (kernel pick, F5, Shift+Enter, table operations). Two engines running when you asked for one was both confusing and a waste of a capacity
• A running Livy session is no longer bypassed to cold-start the local kernel: Ctrl+3 / SELECT-top-1000 previews, table DDL and DML operations, and "Show table version history" each routed to the local engine whenever it was installed, cold-starting it while a Livy session sat idle — and under the new single-engine rule that stray start would also have killed the session. All three now run on whichever engine is actually up
• The engine picker is retired, replaced by a T-SQL toggle: with one engine running, "which engine" stopped being a choice — it is whichever kernel is up, else the machine default. What remains a real choice for .sparksql tabs is the read-only T-SQL endpoint, which is not an engine but a different view of the lakehouse (what DirectLake serves), so the status-bar item is now a two-state toggle. The command id is unchanged, so keybindings and muscle memory survive. It is also finally gated on a SQL tab being focused and the connection having an endpoint — the old item claimed that gate in its comment but showed in every editor
• One state grammar across both engine status-bar items: outline icon at normal foreground = stopped and clickable, yellow spinner = starting, green filled = running, red = failed. Prominence now encodes state, not engine identity. The Livy item's permanent yellow fill when no session was running is gone — it was on screen constantly, decayed into banner blindness, and out-shouted whichever engine was actually working
• The local engine is discoverable: its status-bar item is no longer hidden when the engine isn't installed on the machine — it shows dimmed, and clicking it runs Setup local engine. Hiding it meant word of mouth was the only way to learn the local engine existed
• Engine naming: every user-facing label now reads Local engine rather than "Spark (Local mode)" — it is not Spark, it is our own engine with a PySpark-compatible surface, and calling it Spark misattributes both its wins and its gaps
• New automated gate for the Python shim: python -m pytest python/tests -q (12 tests, pure stdlib, no pyspark needed). Run it after any shim change — a syntax error in the shim otherwise soft-fails at kernel init as a single log line that silently disables notebook.run
• Files touched: python/notebookutils_shim.py, python/tests/ (new), src/extension.ts, src/sparkSessionManager.ts, src/workspaceNotebookResolver.ts, src/aileronKernelConfig.ts, src/aileronController.ts, src/setupLocalSpark.ts, package.json, README.md, CLAUDE.md

2.1.5

• The local kernel no longer starts itself behind your back: with a Livy session running, opening a notebook could have VS Code auto-select (or restore) the Local engine kernel, and our selection handler booted the local kernel as a side effect — so two engines ended up running when you had asked for one. Notebook Preferred is now revoked from the local controller while a Livy session is up, and warming up on selection is skipped when a selection looks like VS Code restoring a remembered binding (arriving within 5s of the notebook opening) rather than a pick you just made. The heuristic's failure modes are both harmless — one warm-up we didn't need, or one cold start on a cell that could have been warm — since running a cell always calls ensureReady first
• Engine picker: three-way, and visible whenever there is a choice: the .sparksql status-bar engine item now opens a quick pick over Spark SQL (Livy API) · Local engine · T-SQL endpoint instead of flipping between two. It is shown based on availability rather than on both kernels being green: Livy when a session is running, Local mode whenever the engine is installed (picking it is how you ask for it — F5 starts it lazily, exactly as Spark starts a session at execution time), T-SQL whenever the active connection has an endpoint. The old rule hid the item in the now-common case where only one kernel is actually up, and T-SQL had been unreachable from it since the local engine arrived
• A manual engine choice now survives kernel state churn: the pin set by the picker was cleared on any kernel state change — harmless while the picker only offered the two kernels, but it silently reverted a T-SQL choice the moment a Livy session went idle→busy→idle mid-query. The pin now holds as long as the engine it named is still on offer
• Files touched: src/aileronController.ts, src/extension.ts, package.json, README.md, CLAUDE.md

2.1.4

• Local mode replaces the External VM, and is now the default engine: the native engine that shipped opt-in on an Azure dev VM in v2.1.0 now runs on your own machine — a loopback Jupyter server started by the extension, reading OneLake directly under the active connection's identity. No VM, no SSH tunnel, nothing to provision. The VM's entire measured advantage was ~2s on the first touch of each table (a Delta-log + token cost that amortizes) against 30+ per-customer machines to provision, grant Managed Identity, patch and keep on a matching engine build — and the laptop is the faster machine (16 physical cores vs 4). Removed: sshTunnel.ts, vmManager.ts, devVmConfig.ts, the Start/Stop VM commands, the SSH memory poller and the externalVm* context keys. A customer who mandates that data stay in-region uses Livy (remote Fabric Spark), which needs no infrastructure of ours
• Lakehouse Studio: Setup local engine (palette): builds the kernel environment once per machine — a venv at %LOCALAPPDATA%\EcitLakehouseStudio\venv, deliberately outside every repo so one environment serves all customers — then installs the engine, the bundled platform baseline, and this workspace's optional setup/kernel-requirements.txt overlay, and verifies by importing what the kernel actually imports rather than trusting pip's exit code
• The engine ships separately from the extension: downloaded from a private feed with your own Azure sign-in and SHA-256-verified before install, with a once-a-day check that offers Update engine when a newer build is published. New machine-scoped settings: aileron.wheelFeed, aileron.wheelTenantId, aileron.wheelSource (a local wheel for engine development — deliberately wins over the feed), aileron.pythonPath
• Engine naming: every user-facing label now reads Local engine (status bar, kernel picker, engine toggle, results footer) or "the local engine" — the internal component name no longer appears in any notification, tooltip, progress title or error, so nobody has to learn it to read a message in front of a customer. All labels route through one helper, since these names previously lived in ~20 literals across four files and every one was wrong the day the name changed. Setting keys, the controller id, output-channel names and log lines keep the real name on purpose (renaming keys would orphan configured values, and support conversations need the unambiguous name)
• Lost query results now fail loud instead of rendering as "0 rows" — the most serious bug fixed in this release. jupyter_server rate-limits kernel output and drops (not truncates) any message over budget, so a ~2.6 MB result worked while a ~2.7 MB one returned nothing and the grid showed "Showing all 0 rows" — indistinguishable from an empty table. The limit is now disabled at server startup (our payloads are bounded by construction via the 1000-row display cap), and on top of that a missing or unparseable result payload is reported as an explicit error — in the local SQL path, the notebook cell renderer, and the Livy notebook path — instead of empty rows or a grid still holding the previous query's data
• Local writes are attributed to the connection's identity everywhere: the .sparksql tab and tree-view paths never passed the connection to the engine's connect call, so they silently fell back to your personal az login — a service-principal connection's writes were committed as the consultant instead of the SP, and an SP-only lakehouse failed on those paths while notebooks worked. All three entry points now follow the connection
• Python environment hygiene for every interpreter we spawn: PYTHONUTF8=1, because an unqualified open() defaulted to UTF-8 on the Linux VM and on Fabric but to cp1252 on a Danish Windows laptop — æøå files would mojibake or raise UnicodeDecodeError in extracts that "worked yesterday"; and inherited PYTHONPATH/PYTHONHOME are dropped, since an Anaconda value is prepended to the venv's sys.path and the kernel then imports a foreign pyarrow and dies with "DLL load failed"
• Setup robustness: the py launcher's -3 selector now travels with every invocation (without it py.ini/PY_PYTHON silently picks the wrong Python); a configured interpreter path that neither exists nor looks like a venv is no longer "repaired" in place (two blind dirnames on C:\Python312\python.exe derived C:\, and setup would then build a venv into the drive root); a missing bundled requirements file now fails at that actual cause instead of marching on to a misleading "cannot import pandas"; and a failed download shows its message modally with an explicit "Pick a wheel file…" rather than dropping you into a silent file dialog
• Engine-download errors say what to do: a sign-in that exists only in the customer tenant is named as such (with the guest-account fix) instead of an unreadable AggregateAuthenticationError; a missing Azure CLI, corporate TLS inspection (NODE_EXTRA_CA_CERTS) and a black-holed proxy each get their own message; whole-request timeouts (60s for the manifest, 10 min for the ~50 MB engine) stop a hung download from freezing the progress notification; and the 403/404 message now lists storage-firewall IP allow-listing as a third cause that looks identical to the other two
• No automatic git pull at startup: a silent pull on activation could hang on a credential prompt, and pulling is the user's call. The long-inactivity notification now offers a plain Reload (was "Pull & Reload")
• No first-run setup prompt: local mode is rolled out person-to-person, so activation stays silent — the pointer to Setup local engine appears when someone actually tries to start the local kernel without an engine installed
• Quieter local-kernel status bar: the item stays hidden until the engine is actually installed (so a colleague is never offered a kernel that cannot start), and its colours are deliberately subdued — Livy is the production engine and should be the item that draws the eye. Cold start narrates its phase; a start failure offers Setup local engine / Retry. Machine memory is read from Node instead of polled over SSH
• No kernel memory-pool overrides: the previous greedy pool at 0.7 was sized for the VM's 58 GiB cgroup backstop. A laptop has no such backstop — it pages — so we now inherit the engine's own conservative default
• Packaging hygiene: .claude/, CLAUDE.md, __pycache__, *.pyc and stray *.vsix files are excluded from the published extension; pyarrow and tzdata are now declared explicitly in the kernel baseline (both arrive transitively today, the verify step imports both, and tzdata is the timezone database Windows does not ship)
• Fixed: the palette entry for the local kernel still read "Aileron kernel: start / stop / restart" — it now matches the rest of the UI
• Files touched: src/aileronKernelManager.ts (renamed from vmKernelManager.ts), src/aileronKernelConfig.ts (new), src/localKernelServer.ts (new), src/kernelTransport.ts (new), src/setupLocalSpark.ts (new), src/engineFeed.ts (new), src/aileronController.ts, src/extension.ts, src/fabpyNotebookController.ts, src/jupyterWsClient.ts, src/notebookQueueManager.ts, src/notebookSequenceRunner.ts, src/dataFrameViewerProvider.ts, src/sparkSessionManager.ts, src/sparkSqlCodeLensProvider.ts, python/kernel-requirements.txt, .vscodeignore, package.json, README.md, CLAUDE.md; removed src/sshTunnel.ts, src/vmManager.ts, src/devVmConfig.ts, UX_IMPROVEMENTS_PLAN.md

2.1.3

• Grid range selection (SSMS-style): drag a rectangular cell range in the results grid — Shift+click / Shift+arrows to extend, Ctrl+arrows to jump to data edges, click/drag the # gutter to select whole rows — and Ctrl+C copies the range with column headers (with the familiar green flash). Required upgrading the bundled Tabulator 5.5.0 → 6.3.1; single- and multi-result grids share the range options/CSS so they can't drift
• Restore table to version…: right-click a table → pick a version from its Delta history (newest first, current version marked) → one confirm modal (with an extra PRODUCTION warning line on PRD connections) → RESTORE TABLE … TO VERSION AS OF n via Livy. On success the table row and intellisense refresh, since a restore can change the schema. Spark/Livy-only — requires a running Spark session; a restore writes a new commit so it's undoable by restoring again
• Ctrl+3 previews temp views: Preview Table now also works on bare temp-view names — a dot-less identifier previews via SELECT * FROM <view> in the live Spark/Aileron session (T-SQL tabs fall back to Spark, since the SQL endpoint has no temp views)
• Aileron cold start narrated: the first VM query of the day now shows where in the boot it is (Waiting for SSH → Starting Aileron kernel → Connecting to Aileron) in the progress toast, results panel, and status bar — with a "first query of the day, ~40s" heads-up when the VM isn't running yet. If the kernel doesn't come up, the error offers Start VM / Retry buttons instead of a bare message
• Aileron per-query cache metrics: the engine's metrics envelope (total/planning/execute ms + per-table cache source: resident/refresh/fill_pending/remote) is captured per query, logged in the "External VM Kernel" output channel's PERF line, and attached to the result. The kernel memory pool fraction was lowered 0.8 → 0.7 to leave headroom for the engine's resident table cache
• Results footer badge: Aileron results now read "via Spark (Local VM)" (matching the v2.1.1 engine rename; was "via Aileron")
• Clearer mid-run session death (Livy): if the Fabric Spark session dies mid-statement (404 on the poll), the extension tears the session down and reports "Fabric Spark session ended during execution…" instead of a raw polling error; statement errors are headline-first (ename: evalue on line 1, traceback below)
• Cleanup round: removed the long-deprecated autoLoadModules setting, dead HC-era code and stale comments, and unused dependencies (uuid, hyparquet, @azure/storage-blob); the duplicated table/column diagnostic logic in the Python and .sparksql providers was consolidated into shared sqlParser.ts helpers
• Files touched: src/dataFrameViewerProvider.ts, src/extension.ts, src/vmKernelManager.ts, src/sqlParser.ts, src/sqlDiagnosticProvider.ts, src/sparkSqlDiagnosticProvider.ts, src/sparkSqlCompletionProvider.ts, src/sqlSchemaRegistry.ts, src/lakehouseProvider.ts, src/abfssClient.ts, src/utils.ts, src/fabpyNotebookController.ts, src/schemaComparisonProvider.ts, src/sparkSessionManager.ts, resources/vendor/* (Tabulator 6.3.1), package.json, README.md, CLAUDE.md

2.1.2

• Faster runMultiple progress grid (Livy): streaming statement polling now runs at 1s (was 2s) and the first poll fires 0.5s after submit (was 2s), so the notebook.run / runMultiple progress grid refreshes roughly twice as fast and the first grid appears ~1.5s sooner. The Python-side emit throttle (~1.5s) is unchanged, so the large-DAG stdout-flood protection from v2.0.0 is untouched
• Queue counter recalculates on mid-run enqueue: queuing another notebook from the tree Play button while a run is in progress now grows the overall cell total immediately (e.g. 2/6 → 2/12) instead of waiting until that notebook starts. Cell counts are computed at enqueue with the same parser the runner uses, and reconciled at start if the notebook file changed in between
• Files touched: src/sparkSessionManager.ts, src/notebookQueueManager.ts, src/notebookSequenceRunner.ts, src/dataFrameViewerProvider.ts, python/notebookutils_shim.py (comment only), package.json, README.md, CLAUDE.md

2.1.1

• Engine rename (UI-only): the two engine status-bar items now read Spark (Livy API) and Spark (Local VM) (the underlying engine is still Aileron), with the env/memory suffix in a trailing paren — Spark (Livy API) (DEV), Spark (Local VM) (3.2 / 64 GB). The .sparksql engine toggle shows Spark SQL (Livy API) / Spark SQL (Local VM). Internal ids, context keys, output channels, and Python error names keep the real "Aileron" name
• Aileron cancel that actually cancels: Escape (on a .sparksql tab) and the notebook Stop button now interrupt (SIGINT), wait a 1.5s grace so pure-Python work cancels cheaply and keeps its session state, then hard-restart the kernel when a query is still running — aileron's native layer ignores SIGINT mid-query, so a cancel previously hung 30–60s. The restart reuses the live SSH tunnel (~2–4s, no cold start); the killed query renders "Cancelled", and a hard cancel warns that temp views/variables were cleared
• Engine-aware tree Run / queue: the notebook-tree Play button (and its queue) now run on the active engine instead of Livy only — Spark wins when both engines are green (a manual engine-toggle pin is honored); otherwise whichever is running. notebook.run / runMultiple notebooks stay blocked from the tree on both engines (run those from the editor)
• Transactional DML result grid (both engines): UPDATE / MERGE / DELETE now render in the normal result grid — operation | rows_affected for a single write (Spark MERGE adds its updated/deleted/inserted breakdown), or one row per statement for an all-write batch — instead of the centered checkmark. Spark now captures Delta's metrics DataFrame it previously discarded; T-SQL is read-only and unaffected
• Fresh column metadata: expanding a table or schema in the Lakehouse Explorer now revalidates it in the background against the OneLake Unity Catalog API (on-demand, 60s throttle, offline-safe) so a column/table added elsewhere appears without a manual refresh; "SELECT top 1000 rows" on Spark/Aileron fetches live columns (with a status-bar spinner) like the T-SQL preview already did. Credential instances are cached per auth identity so these calls stay sub-second
• Snappier result grid: the results panel keeps its per-tab grids alive when hidden (retainContextWhenHidden), and tab switches now activate the cached pane without re-shipping row data — switching between many result tabs is instant. Each tab shows its own execution duration in the footer
• Honest execution feedback: the indeterminate "Executing…" spinner is now a live Phase · mm:ss timer; the Livy path reports Submitting → Queued → Running
• Grid libraries bundled locally: Tabulator and SheetJS now ship in the extension (resources/vendor/) instead of loading from a CDN, so the grid and Excel export work offline / behind corporate proxies; multi-result grids are virtualized
• Files touched: src/vmKernelManager.ts, src/aileronController.ts, src/notebookQueueManager.ts, src/notebookSequenceRunner.ts, src/dataFrameViewerProvider.ts, src/lakehouseProvider.ts, src/authProvider.ts, src/connectionManager.ts, src/sparkSessionManager.ts, src/extension.ts, resources/vendor/* (new), package.json, README.md, CLAUDE.md

2.1.0

• External VM (Aileron) execution engine (experimental, opt-in): a third backend alongside Livy (Spark) and the SQL Endpoint (T-SQL). Notebook cells and .sparksql tabs run on a native Aileron engine hosted on an Azure dev VM, reading OneLake directly via managed identity — no Spark cluster, no Livy session. Off by default; the feature is purely additive and the existing Spark/T-SQL paths are unchanged when it's disabled
• Opt-in per workspace: enabled by a workspace-local setup/external_vm/dev_vm.json with { "enabled": true, "sshHost": "…" } (not bundled in the extension, so the published build ships with VM mode off). Connects over an SSH tunnel to a token-less Jupyter kernel on the VM
• Notebook "External VM" kernel: appears in VS Code's Select Kernel picker and becomes the preferred kernel for Fabric notebooks when VM mode is on. SQL cells, %%sql, and PySpark-style code (spark.sql, df.write, from pyspark.sql import …) run unchanged via aileron's aileron.spark compatibility layer; local utility/*.py imports are inlined (the VM has no Fabric Files mount); notebook.run / runMultiple reuse the same registry + live progress grid as the Livy path
• .sparksql Aileron engine: the per-tab engine toggle becomes a 3-way cycle Spark → Aileron → T-SQL; results render in the DataFrame Viewer with a "via Aileron" badge. Display is capped at 1000 rows (OOM guard — the engine materializes the whole result before the grid slices it)
• Azure VM lifecycle + health: Start / Stop (deallocate) the dev VM from the connection's right-click menu; a dedicated "Aileron" status-bar item shows kernel state and whole-VM memory. The VM power state is polled (on activation, every 60s, and on window focus) so the UI stops showing the kernel as runnable after Azure deallocates the VM out-of-band
• Files touched: src/devVmConfig.ts, src/vmKernelManager.ts, src/aileronController.ts, src/vmManager.ts, src/sshTunnel.ts, src/jupyterWsClient.ts (all new), src/extension.ts, src/dataFrameViewerProvider.ts, src/queryHistory.ts, src/sqlParser.ts, src/sparkSqlCompletionProvider.ts, src/sparkSqlCodeActionProvider.ts, src/sparkSqlCodeLensProvider.ts, python/notebookutils_shim.py, package.json, README.md, CLAUDE.md

2.0.5

• Pipeline Explorer (new GIT-mode tree view): scans the repo for *.DataPipeline folders (mirroring the Notebook Development (Git) view) and offers exactly two actions per pipeline — Run pipeline… and Cancel pipeline. Execution is connection-aware: it resolves the pipeline's Fabric item id in the active connection's notebook workspace (getNotebookWorkspaceId) and runs there, so the same tree row runs DEV on a DEV connection and PRD on a PRD connection
• Parameter prompting from the local repo: if pipeline-content.json declares properties.parameters, Run prompts for each value with the repo defaultValue pre-filled (e.g. "All"), coerces by declared type (int/float/bool/array/object/string), and passes them as executionData.parameters. Parameter-less pipelines run immediately
• Live execution monitor (webview): opens on Run and polls Get pipeline job instance + Query activity runs every 4s, rendering a status grid styled like the runMultiple progress grid — columns Activity · Status · Run start (Copenhagen time) · Duration · Error. ForEach iterations collapse under one group row; Invoke-Pipeline activities expand (one level, on demand) to show the child pipeline's activities first, with raw Input/Output demoted to secondary collapsible toggles; click any activity to inspect its input/output JSON
• Honest cancel state: clicking Cancel shows "Cancelling…" and keeps polling until Fabric actually reports the terminal status (the job-level status lags after a cancel), then flips to the real "Cancelled" and freezes the elapsed timer — no more a spinner stuck on "InProgress" forever
• Elapsed/Run-start parsed as UTC: Fabric's *Utc timestamps sometimes arrive without a Z; they're now normalized to UTC so the elapsed counter starts near 0 instead of being skewed by the local timezone offset
• Notebook Development (Remote) removed: the unused REMOTE tree view, its provider, and all remoteNotebook.* commands/menus/welcome entries are gone
• "Connect to Azure…" renamed to "Login to Azure…": and the best-effort az account set --subscription step was dropped from the login (capacity/ARM calls embed the subscription id in their URLs and the token is tenant-scoped, so it added latency without value); the post-login toast no longer lingers with a misleading "Refreshing…"
• Files touched: src/pipelineProvider.ts (new), src/pipelineMonitorProvider.ts (new), src/fabricApiClient.ts, src/extension.ts, src/connectionManager.ts, src/remoteNotebookProvider.ts (deleted), package.json, README.md, CLAUDE.md

2.0.4

• %pip install works from the notebook tree "Run": the tree/queue runner now converts %pip install / !pip install to a subprocess.check_call just like the in-editor controller does (Livy can't parse pip's plain-text stdout). Previously a queued/tree-run notebook sent the raw magic to Livy and failed with "unknown command". Both paths now share a single convertPipInstallMagic() helper so they can't drift
• DataFrame Viewer shows the literal string "None" instead of masking it as NULL: the viewer used to render a string-valued "None" as a grey NULL, which made WHERE col IS NULL look broken (it returned nothing) while WHERE col = 'None' worked. The value is genuinely the string "None" in Spark (it matches server-side), so the viewer now shows it verbatim; only real null/undefined render as NULL
• Spinning refresh icon while a table/Tables refresh runs: clicking Refresh on a table row now spins that row's icon ($(sync~spin), always visible — not just on hover) until the column sync completes; the Tables-category refresh button spins too. Only the clicked table's row spins, not every table
• Cloned notebooks no longer produce a phantom per-cell METADATA diff: opening + saving a duplicated notebook used to reconstruct every cell's # METADATA block in a canonical form, diffing against a source whose block was formatted differently. NotebookSerializer.serialize() now preserves each cell's original METADATA block verbatim (falling back to reconstruction only for genuinely new cells, guarded by a language match), so an unchanged round-trip is byte-stable
• "Connect to Azure…" command on every connection: a new right-click action runs az login --tenant <tenantId> (pinned to the connection's tenant, falling back to plain az login when none is set) and then best-effort az account set --subscription <id> so the CLI state and DefaultAzureCredential token acquisition match the connection
• SQL Endpoint "SELECT top 1000" resolves live columns (SSMS-style): the T-SQL preview now queries INFORMATION_SCHEMA.COLUMNS over the pooled tedious connection for the real, current column list instead of trusting the possibly-stale schema_reference.json, falling back to the cache on any error. The Spark preview is unchanged (no TDS connection)
• Files touched: src/utils.ts, src/fabpyNotebookController.ts, src/notebookSequenceRunner.ts, src/dataFrameViewerProvider.ts, src/lakehouseProvider.ts, src/notebookSerializer.ts, src/connectionManager.ts, src/sqlEndpointExecutor.ts, src/extension.ts, package.json, README.md, CLAUDE.md

2.0.3

• Drag table/column names from the Lakehouse Explorer into the editor: dragging a table node inserts schema.table (e.g. bronze.customer); a column node inserts the bare column name. Drops land at the mouse position in .sparksql tabs and notebook cells. Implemented via a private-MIME TreeDragAndDropController + DocumentDropEditProvider so the inserted text is exact (no text/uri-list table: prefix from the node's decoration resourceUri). Columns drop directly; dotted schema.table shows VS Code's "hold Shift to drop" hint and inserts on Shift-drop
• "+" Add Connection now bootstraps local_development\: previously the Add-Connection command bailed out when the folder was missing — but the folder is only created on connection save, so the very first connection could never be created (chicken-and-egg). The command now only requires an open workspace folder; saveNewConnection() creates the folder as before
• Queued notebook runs inline non-nb_ function modules: the tree Play/queue runner only inlined from <name> import * when <name> had the nb_ prefix, so a renamed from dataplatform_functions import * reached Livy raw and failed with ModuleNotFoundError. It now inlines any from <name> import * resolvable to a local utility .py / .Notebook (honoring the utilityModulesPath setting) and leaves genuine stdlib imports (e.g. from typing import *) untouched — matching the in-editor controller
• Files touched: src/lakehouseDragController.ts (new), src/extension.ts, src/connectionManager.ts, src/notebookSequenceRunner.ts, package.json, README.md, CLAUDE.md

2.0.2

• Notebook saves no longer create phantom git diffs: NotebookSerializer.serialize() now ends notebook-content.py with the final # META } line + a single newline (no trailing blank line), matching what Fabric's git integration writes today. Earlier versions appended an extra blank line (Fabric used to write one), which produced a one-line diff on every Fabric re-sync. Verified byte-identical round-trip on real Fabric-synced notebooks
• Spark "SELECT top 1000 rows" preview forces the Spark engine: the right-click Spark preview now sets the new .sparksql tab's engine to Spark (mirroring the SQL-Endpoint variant), so backtick-quoted preview SQL no longer gets routed to the T-SQL endpoint when T-SQL was the last-used engine
• T-SQL alias. column completion works with bracketed identifiers: parseTableAliases/parseAllTables now match bracketed ([schema].[table]) and backticked table references and strip the delimiters, so column completions appear immediately after typing alias. in T-SQL FROM [bronze].[table] a (previously empty until a character was typed)
• No auto-save on Run All: running all cells in a notebook no longer force-saves the file
• local_development/ is created lazily: the folder is now created on first connection save (the moment we know it's Fabric work) instead of on every activation — non-Fabric repos stay clean. All readers already tolerate its absence
• Files touched: src/notebookSerializer.ts, src/extension.ts, src/sqlParser.ts, src/fabpyNotebookController.ts, src/connectionManager.ts, package.json, README.md, CLAUDE.md

2.0.1

• "SELECT top 1000" expands all columns (SSMS-style): the right-click preview commands now emit every column explicitly (one per leading-comma line, alias-prefixed) instead of SELECT a.*, with engine-aware quoting via quoteColumnForScript — T-SQL brackets every column and the schema/table (FROM [bronze].[customer] a), bulletproof against reserved-keyword columns like [open]/[lineNo] with no keyword list to maintain; Spark backticks only names that need it. Falls back to a.* when columns aren't cached, and the cursor lands on the blank line below the statement
• T-SQL "fetch up to 100k" via the Count-all-rows toggle: for a T-SQL single SELECT, turning the DataFrame Viewer "Count all rows" toggle ON raises the fetch cap from 1,000 to SQL_ENDPOINT_MAX_ROWS = 100000 (with the background COUNT_BIG(*) still filling the true total → Showing 100.000 of X rows). Spark and multi-statement runs stay at 1,000
• Count-all-rows toggle resets OFF on fresh starts: opening a new .sparksql tab or running a "SELECT top 1000" preview (either engine) forces the toggle back OFF, so previews/new tabs always default to the fast capped path
• Auto-size columns button: a new first button in the DataFrame Viewer toolbar sizes every column on the active tab to its widest visible value — the same effect as double-clicking a column border, applied to all columns at once
• Auto-load utility modules is folder-only: at Spark session startup the extension always scans lakehouseStudio.utilityModulesPath (default utility/) and loads every top-level .py (skipping __init__.py and _/. prefixes); a missing folder is a clean no-op. The legacy comma-separated lakehouseStudio.autoLoadModules setting is deprecated and ignored
• Files touched: src/extension.ts, src/sqlParser.ts, src/dataFrameViewerProvider.ts, src/utils.ts, package.json, README.md, CLAUDE.md

2.0.0

• Connection row text reworked: the tree description now reads workspace | lakehouse | <capacityState> (<SKU>) — e.g. ecit-dpf-store-dev | lhEwh | Active (F8), or … | Paused (F8) / … | Resizing… (F8). The [DEV]/[PRD] tag is dropped from the row (env shows via icon color + tooltip, and conventionally in the connection name); inactive rows and active rows without capacity info show just workspace | lakehouse. Friendly-name placeholder updated to E.g. "Philipson Wine [Dev]"
• "Default Lakehouse…" uses the active connection: the picker now scopes strictly to the active connection's lakehouse workspace (previously fell back to the Dev connection, which could surface another customer's lakehouses). A modal hard-stop fires when the notebook already has more than one known_lakehouses (set those in the Fabric portal), and a modal warning fires before pointing a notebook at a production connection. The notebook's existing kernel_info is preserved, and only the top # META block is rewritten (per-cell metadata untouched)
• Special-character columns are auto-quoted, engine-aware: column names that aren't plain identifiers (contain ., -, æøå, whitespace, or a leading digit) are wrapped on intellisense insert and alias.* expansion — backticks for Spark SQL (`Art.nr`), square brackets for T-SQL ([Art.nr]). .sparksql providers follow the focused tab's engine; Python spark.sql() providers always use backticks
• Parallel multi-statement Spark SQL: multiple SELECTs in a .sparksql file now run concurrently inside one Livy statement via an in-session ThreadPoolExecutor (up to 5), results rendered in original editor order. A single statement is unchanged (keeps its async row count)
• runMultiple stable on large DAGs: ~80-notebook DAGs previously crashed the Livy session (json4s MappingException, session alive but mute). Fixed on both size axes — the notebook registry is chunked across separate Livy statements (256 KB budget) instead of a single ~1 MB payload, and the shim's progress markers are throttled (≤1 emit/~1.5 s + forced final state) instead of re-serialising all activities on every transition
• Files touched: src/lakehouseProvider.ts, src/connectionManager.ts, src/notebookSerializer.ts, src/sqlParser.ts, src/sqlCompletionProvider.ts, src/sparkSqlCompletionProvider.ts, src/sqlCodeActionProvider.ts, src/sparkSqlCodeActionProvider.ts, src/extension.ts, src/fabpyNotebookController.ts, python/notebookutils_shim.py, README.md, CLAUDE.md

1.9.9

• Cold start drops from ~3 min to ~8-12 s: HC Livy (/highConcurrencySessions) replaced with the legacy /sessions endpoint + a minimal POST body ({name} + optional spark.fabric.environmentDetails). Fabric attaches the standard-size starter pool when no extra conf keys are sent — same path the in-portal notebook UX uses. One Spark session per connection — no 5-REPL subsessions, no auto-spin, no (n/5) status-bar counter, no sessionTag. The 5 spark.sql.parquet.*RebaseMode* / spark.python.worker.faulthandler.enabled conf keys that v1.9.x sent are gone (they disqualify starter pool attach and were not load-bearing for our customer code). Public API of SparkSessionManager preserved (replId == sessionId 1:1) so callers don't change
• Multi-SELECT in .sparksql is sequential: each statement runs to completion on the single Spark session before the next is submitted. Matches the "multiple notebook cells run one after another" mental model. notebookutils.notebook.runMultiple stays parallel (ThreadPoolExecutor in the shim with predecessor-event gates)
• Capacity SKU on the connection row: active connection description now reads F4 — Active [DEV] — lhEwh. getCapacityStatus() parses data.sku.name from the Azure ARM GET. While a scale is in flight the row reads F8 — Active (Capacity: Resizing…) [DEV] — lhEwh
• Scale capacity… right-click command: new entry next to Resume/Pause. Picker offers F2 / F4 / F8 (current size hidden); PUT to /Microsoft.Fabric/capacities/{name} does a read-modify-write so admins/location/tags survive. Tree flips to Resizing… immediately; polled at 15s / 45s / 90s. Allowed in both Active and Paused state. Re-entry blocked while a scale is in flight via isScaleInFlight()
• "Refresh All Columns (Full Rebuild)" moved to command palette only: rarely needed (auto-runs on first connection save, background sync handles steady state). Now palette-only as Lakehouse Studio: Refresh All Columns (Full Rebuild) for emergency rebuilds. Handler accepts palette invocation (no tree-item arg required — falls back to the active connection)
• Env picker is tighter: when the workspace has exactly one Fabric environment, the session auto-attaches to it with no picker. When two or more envs exist, the picker shows only those envs — "Standard" is no longer offered as an override (it remains the implicit default only when the workspace has zero envs)
• Status bar polish: $(circle-filled) green when idle, $(circle-outline) yellow when no session. Click with no session skips the one-entry menu and fires the env picker directly
• Files touched: src/sparkSessionManager.ts (substantial rewrite — legacy /sessions model, single-session-per-connection, tighter env picker), src/capacityManager.ts (SKU on CapacityStatus, new scaleCapacity() + in-flight guard), src/lakehouseProvider.ts (SKU prefix on tree description), src/extension.ts (Scale capacity command + handler, sequential multi-SELECT, palette-only refresh-all-columns), src/models.ts ('Resizing' state + skuName field), src/fabpyNotebookController.ts (string renames), package.json (new command, removed orphan debug command, menu entry), README.md, CLAUDE.md
• Removed: src/hcSessionSpike.ts + its lakehouseStudio.debug.testHcSession command def; src/sessionStartupExperiment.ts + its 7-variant picker. Both were diagnostic harnesses for the HC era that are no longer needed

1.9.8

• Danish characters in HC session names fixed: Fabric's HC Livy API (POST /highConcurrencySessions) returns HTTP 400 when name or sessionTag contain non-ASCII. Developers with æ/ø/å in their Windows username (e.g. Søren) or Fabric environments named with Danish letters (e.g. Miljø-DEV) could no longer acquire a session. New sanitizeForFabricSessionName() in utils.ts maps æ→ae, ø→o, å→a (and capitals) and strips any remaining non-[A-Za-z0-9_-] chars as a safety net. Applied to firstName, envText, envName before they flow into sessionTag and hcPostName in sparkSessionManager.ts. ASCII-only developers see byte-identical names, so existing sessionTag packing keeps working

1.9.7

• Subsession runaway fixed: starting a session no longer spawns endless subsessions. _acquireRepl now decides create-vs-join on an atomic re-read of the group (closing the concurrent-first-acquire race that created multiple groups, each auto-spinning), background auto-spin acquires are join-only (can never resurrect/spawn a group), and a disposed flag makes in-flight acquires abort instead of repopulating after an End
• End during cold-start now works: the underlying sessionId is captured onto the group as soon as it appears during polling (before replId), so "End main session" can DELETE it mid-startup; the real hcId is stamped onto the placeholder so ending a starting subsession no longer 400s; and the HC poll aborts within ~3s when the group is disposed
• Smart keepalive heartbeat: one timer per group fans a SELECT 1 to every idle subsession every 10 min (Fabric enforces a fixed ~20-min idle timeout per subsession attachment and ignores heartbeatTimeoutInSecond). The beat window is livySessionKeepAlive − 20 min measured from the last real execution, so an active session never expires and an idle one reaps ~livySessionKeepAlive min after its last run. Default livySessionKeepAlive raised to 120 min
• Status bar (n/x) after connection switch: the in-use count now aggregates across all groups' live subsessions, so switching connection (which retargets subsessions via env vars) keeps the count instead of dropping to a bare "Spark"
• DataFrame row count is opt-in: Spark queries default to "Showing 1000 rows" (the slow background count no longer auto-runs). A "123" toggle in the DataFrame Viewer title bar (green when on) enables the total count for subsequent queries; OFF at every activation. T-SQL still always counts
• DataFrame Viewer fixes: dotted column names like Art.nr render correctly (Tabulator nestedFieldSeparator: false — no more phantom NULLs); multi-result grids match single-result font/row size; multi-result mode gains a per-result Copy button and a right-click "Copy Cell" with the green-flash feedback; the single-result Copy button now flashes green too
• Connection edit hardening: switching a connection from Service Principal to Interactive now wipes the stale tenantId/clientId and the SecretStorage secret, and the access-token cache is dropped on any connection change. First-time connection save no longer blocks the dialog on the initial intellisense build (fire-and-forget with a non-modal toast)
• Per-statement timeout default raised from 60 to 180 min (livyStatementTimeout); startup-failure red status-bar flash extended to 5 s

1.9.6

• Multi-lakehouse writes via Fabric native 4-part naming: new python/spark_sql_shim.py wraps spark.sql and rewrites every schema.table reference to Fabric's native 4-part identifier `<workspace>`.`<lakehouse>`.schema.table based on FABRIC_WORKSPACE_NAME / FABRIC_LAKEHOUSE_NAME env vars. Switching the active connection in the tree view re-injects those env vars into every subsession in parallel — subsequent queries write to the new target lakehouse without restarting Spark. Works universally (SELECT, INSERT, MERGE, ALTER, OPTIMIZE, DROP TABLE — including the silent-no-op DROP that earlier abfss-path approaches couldn't solve). Tokenizer-based; 3-part and 4-part user identifiers pass through unchanged
• ABFSS path qualifier: new python/spark_path_shim.py monkey-patches DataFrameReader.csv/json/orc/text/load/parquet and DataFrameWriter.csv/parquet/json/orc/text/save so relative Files/... and Tables/... paths auto-qualify to the active connection's abfss://<ws>@onelake.dfs.fabric.microsoft.com/<lh>/.... Customer notebooks no longer hand-build ABFSS URLs. Pass-through for already-qualified abfss://, s3a://, https://, POSIX paths
• High Concurrency Livy sessions: extension now exclusively uses Fabric's HC Livy API (/highConcurrencySessions), packing up to 5 isolated REPLs ("subsessions") onto one underlying Spark session via sessionTag. Each subsession is a separate Python interpreter on the shared Spark JVM — true driver-level parallelism. New python/notebookutils_shim.py is injected per-REPL alongside the spark-sql / path shims. Internal model: Repl (replaces SparkSession) keyed by replId + HcGroup per connection tracking shared livySessionId, env choice, heartbeat timer
• Auto-spin background pre-warming: after the first subsession of a group is ready (~44s Spark cold-start), the remaining 4 subsessions spin up in parallel (~6s each, joining the existing Spark session) under a single notification-area progress bar: Starting subsessions in background… 1/5 → 5/5. Opening a 2nd notebook immediately attaches to a pre-warmed subsession instead of waiting for a fresh one
• One NotebookController per subsession: kernel-picker (top-right of notebook editor) shows DEV / Subsession 1, DEV / Subsession 2, ..., one entry per REPL. New notebooks auto-attach to the lowest-numbered idle subsession via notebook.selectKernel command (affinity alone isn't enough when multiple controllers exist — needs the explicit command). Subsessions are tracked in _notebookKernels: Map<uri, replId> via each controller's onDidChangeSelectedNotebooks so a user's manual kernel switch is respected
• Status bar shows in-use count: Spark (0/5) = all 5 subsessions idle (full capacity), (5/5) = all in use (saturated). Spinner appears when ANY subsession is busy or starting. Aggregated across all subsessions in the active connection's group — no per-subsession "active" concept anymore. Click for management menu sorted by subsession number
• 'End main session' verb: status-bar menu's bottom item fires DELETE /sessions/{livySessionId} against the regular Livy endpoint, terminating the whole Spark backend and cascading server-side to release every subsession. Auto-fires when the last subsession of a group is individually released, so ending subsessions one-by-one ends with a clean Fabric monitor too. Solves the v1.9.5 era issue where releasing all REPLs left the underlying session orphaned until Fabric's idle timeout reclaimed it
• Sparksql late-bind to first-idle subsession: F5 in a .sparksql tab picks the first idle subsession at execution moment (not at tab open). Multiple tabs F5'd in rapid succession spread across distinct subsessions atomically — _executeOnRepl marks repl.state='busy' synchronously before the first await, so call N+1's first-idle scan sees call N's claim
• Multi-SELECT real parallelism (replaces ThreadPoolExecutor): .sparksql with semicolon-separated SELECTs now dispatches each statement to a different idle subsession in parallel via Promise.all(executeStatement(...)), instead of building a Python ThreadPoolExecutor inside ONE Livy statement. Real driver-level parallelism, granular cancel, simpler code. Results render in editor order (not completion order) — fastest-finishing query doesn't jump to top. Spinner notification during execution, no per-statement progress grid. buildSingleSelectCode(sql, limit) extracted as shared helper; inline .count() per query preserves the synchronous total in the multi-result footer
• Multi-execution cancel tracking: _activeExecution? → _activeExecutions: Map<token, {repl, statementId, abortController}> keyed by ${execCounter}-${replId}. Escape during a multi-SELECT batch cancels every in-flight statement at its per-REPL /statements/{id}/cancel endpoint, not just the most recent
• Connection switch blocks on busy subsessions: clicking a different connection in the tree view while any subsession is mid-execution shows Wait for active queries to finish before switching connection. Busy: … instead of silently queueing the env-var re-injection behind the running statement. Clean refusal, user retries
• 5-REPL cap with explicit toast: attempting to spawn a 6th subsession (manually or via auto-affinity) surfaces HC session has max no of attachments. Close a notebook or end an idle session attachment to free one. No auto-eviction, no second HC session spawn — user-controlled
• Per-group heartbeat (one per HC group, not per REPL): one SELECT 1 every 10 minutes per HC group keeps the underlying Spark session alive past Fabric's idle timeout. Picks any idle subsession in the group as the carrier. Previously every REPL fired its own heartbeat, multiplying API churn 5×; new model is one keep-alive per backend. Per-REPL liveness GETs (separate concern — detects server-side reclaim) remain
• Env-init injection per subsession: env vars (FABRIC_WORKSPACE_NAME/ID, FABRIC_LAKEHOUSE_NAME/ID, IS_LOCAL_LIVY_SESSION, KEY_VAULT_NAME) + the three Python shims are injected into each newly-acquired subsession's namespace, because each subsession has its own isolated os.environ. Fabric environment is captured on the HC group at first acquire and reused for subsequent subsessions (HC packing means the underlying session's config is fixed anyway)
• Subsession name in Fabric monitor: HC POST name is HC_<firstName>_<envText>_<fabricEnvName> (e.g. HC_Emil_DEV_env_r2_light) so the monitor entry tells you who started it, which env, and which Fabric environment. sessionTag stays <firstName>_<envText> so packing is by user+env
• Hidden verification spike: lakehouseStudio.debug.testHcSession (Ctrl+Shift+P only — not in any menu) runs a 7-step HC API smoke test (acquire / poll / sessionTag packing / env-var isolation across REPLs / parallel statements / cancel / selective DELETE). Logs structured [HC-SPIKE] lines to the Livy output channel. Useful as an integration smoke test when troubleshooting against any connection
• SQL Endpoint submenu on table right-click: SELECT top 1000 rows defaults to Spark SQL again (v1.9.5 had defaulted to T-SQL when an endpoint was configured). A new SQL Endpoint submenu surfaces the T-SQL variant explicitly — opens a new tab pre-set to the SQL endpoint, instant preview without a Spark session. Future endpoint-specific actions (refresh metadata, T-SQL templates) drop cleanly into the same submenu
• Notebook controller field renames + display tweaks: _sessionControllers / _sessionStates / _sessionConnectionIds / _sessionQueues / _executeHandlerForSession / addSession / updateSessionState / removeSession → _replControllers / _replStates / _replConnectionIds / _replQueues / _executeHandlerForRepl / addRepl / updateReplState / removeRepl. Controller id fabpy-session-<id> → fabpy-repl-<replId>. Display label <env> / Subsession <n> (dropped the redundant "Spark" prefix — kernel picker is already clearly "Spark")
• Subsession terminology in UI strings: REPL internals stay (matches Fabric's API documentation) but all user-facing strings now say "subsession" — kernel picker label, status bar tooltip, menu items, toasts. "REPL" is too jargon-y for the target audience; "subsession" is self-documenting (session-within-a-session). Status-bar menu title: Spark Subsessions (n/5)
• Connection edit dialog shows resolved names: two new readonly fields below the existing connection settings — Workspace Name (auto) and Lakehouse Name (auto) — populated from _resolveDisplayNames on save/lakehouse-pick. Mirrors the SQL endpoint readonly fields styling. Lets users see the human-readable names that env-var injection uses without trawling the JSON
• MismatchedInputException / JsonParseException recovery messaging: when a notebook cell's runMultiple invocation triggers Livy's Jackson parser to choke (stdout overflow on large multi-notebook batches + progress markers), the error message now reads Livy failed to parse the runMultiple response — stdout buffer likely overflowed. Mitigations: End this subsession and start a fresh one before retrying. Reduce batch size — split into 2–3 runMultiple calls instead of one big DAG. Trim notebooks that print a lot to stdout. Non-runMultiple cells keep the existing pip-style guidance
• Files touched: python/spark_sql_shim.py (new, ~700 LOC incl. inline tests), python/spark_path_shim.py (new, ~200 LOC), python/test_big_sql.py (new regression test for ~50 JOIN customer query), python/notebookutils_shim.py (unchanged shape, injected per-REPL), src/sparkSessionManager.ts (substantial refactor — HC endpoints, Repl/HcGroup model, auto-spin, per-group heartbeat, end-main-session, multi-execution cancel, busy-block on connection switch), src/fabpyNotebookController.ts (REPL renames + force-attach via notebook.selectKernel), src/extension.ts (multi-SELECT parallel dispatch, simple spinner HTML, SQL Endpoint submenu wiring, lakehouseStudio.debug.testHcSession hidden command), src/hcSessionSpike.ts (new diagnostic command), src/connectionManager.ts (readonly workspace/lakehouse name fields), package.json, CLAUDE.md

1.9.5

• Fabric SQL Analytics Endpoint as a second execution backend: new src/sqlEndpointExecutor.ts runs T-SQL SELECTs against the lakehouse's SQL endpoint via the tedious TDS driver (added as a pure-JS dependency). Reuses the connection's existing AAD identity (SQL_ENDPOINT_SCOPE = 'https://database.windows.net/.default'). Sub-second results for 1000-row SELECTs against 40M+ row tables — no Spark session needed. One pooled Connection per workspace connection, 45-minute TTL safely under the AAD token lifetime
• Per-tab engine indicator (status bar): left-side item shown only when a .sparksql editor is focused. $(flame) Spark SQL [DEV] (no background) or $(zap) T-SQL [DEV] with yellow statusBarItem.warningBackground. Click to flip the active tab between engines. Per-tab state lives in _engineByUri: Map<fileUri, 'spark' | 'sql'> on the DataFrame viewer. Re-renders on connection switch so [DEV]/[PRD] follows the active connection. Command: lakehouseStudio.toggleSqlEngine
• Engine-aware Run CodeLens: ▶ Run (Spark SQL) or ▶ Run (T-SQL · read-only) above each statement in .sparksql files. Flips immediately when the tab's engine is toggled. Sits inline with the code so the engine + read-only constraint is impossible to miss
• Single fast result path — Parquet + "All Rows" toggle removed: hard cap at RESULT_ROW_LIMIT = 1000 for both engines. executeSelectViaParquet, executeMultiSelectViaParquet, FULL_MODE_HARD_LIMIT, getEffectiveLimitForFullMode, stripLimitZero, all OneLake temp_queries/ write/download/cleanup code, and the fabricDataFrameResults.enableAllRows/disableAllRows commands + menu items + viewer state all deleted (~280 LOC + 2 commands + 4 menu entries). Single JSON path is now the only path
• Async background row count: when displayed rows hit the cap, footer reads Showing 1000 rows (Counting total rows…) and a background spark.sql(…).count() (Spark) or SELECT COUNT_BIG(*) FROM (<sql>) AS _ecit_ct (T-SQL) populates the final total in place. Race-safe across webview boot ordering — latest values ride along on the next dataChunk postMessage. T-SQL count strips trailing ; and any top-level ORDER BY (paren/quote-aware walker) before wrapping; silently bails on parse errors. Spark multi-SELECT computes per-statement count synchronously inside the same Python via .cache() + .count() + .limit(N).toJSON().collect()
• T-SQL multi-statement support: runSqlEndpointBatch now handles semicolon-separated SELECT/WITH statements sequentially on the same pooled connection (TDS can't multiplex). Same multi-result Tabulator view as Spark, with progress grid via __ECIT_PROGRESS__. Non-SELECT statements error before execution (endpoint is read-only)
• Auto-injected TOP 1000 for T-SQL: injectTopLimit(sql, 1000) rewrites a leading SELECT [DISTINCT] to add TOP N so the server stops streaming early. Skips CTE queries (WITH …) and queries that already specify TOP. Client-side row cap still applies either way
• Connection warm-up: warmupSqlEndpoint(connection, mgr) pre-establishes the tedious connection (AAD token + TDS handshake) on extension activation and on every connection switch when hasSqlEndpoint(connection). First query skips the ~300-500ms handshake — feels instant
• Cancel T-SQL queries: Escape (or the existing cancel icon) now interrupts in-flight T-SQL queries by dropping the tedious connection (Fabric SQL endpoint doesn't honor TDS attention reliably). Wired through lakehouseStudio.cancelExecution. Active request tracked per pool entry via activeRequest
• Lakehouse SELECT top 1000 rows right-click defaults to T-SQL: single menu entry — when the connection has a SQL endpoint configured, opens a new tab pre-set to T-SQL engine and auto-runs (sub-second). Falls back to Spark when no endpoint is configured. The earlier separate (SQL Endpoint) variant was removed — one obvious action, matches SSMS. Ctrl+N and other "new SQL tab" entry points continue to default to Spark SQL (no behavioural change there)
• Ctrl+3 "Preview Table" honors the tab's engine: when invoked from a .sparksql tab with engine = T-SQL, routes the SELECT * FROM schema.table preview through the SQL endpoint. Falls back to Spark elsewhere (Python files, notebooks). Backticks stripped from dbo.customer-style selections so Spark-quoted names still work against T-SQL
• Refresh SQL Endpoint metadata (right-click on active connection): new command lakehouseExplorer.refreshSqlEndpoint calls POST /workspaces/{ws}/sqlEndpoints/{id}/refreshMetadata via the existing requestWithLro helper. Reflects recent table/column changes from the underlying lakehouse without restarting anything
• Connection model + Edit dialog: new fields sqlEndpointConnectionString and sqlEndpointId on LakehouseConnection, auto-populated from getLakehouse's properties.sqlEndpointProperties.{connectionString, id}. Both shown as readonly inputs in the connection edit webview, themed to match other fields (var(--vscode-input-foreground/background/border))
• Auto-backfill of SQL endpoint info for existing connections: resolveNamesIfMissing() extended to fill sqlEndpointConnectionString + sqlEndpointId when they're blank. Runs on extension activation and connection switch — older connections created before this feature populate themselves silently on next use, then warm up immediately. No manual re-save needed
• parseConnectionString accepts three formats: bare hostname (what Fabric's "Copy SQL connection string" gives) → database defaults to the connection's lakehouseName; ADO.NET Server=tcp:…,1433;Database=…;; and JDBC jdbc:sqlserver://…. Auth always uses the AAD token; embedded user/password is ignored
• T-SQL type mapping for the grid: describeType() maps TDS column types to Spark-ish display names (Int → IntegerType, NVarChar → StringType, DateTime2 → TimestampType, etc.) so the column type column in the result grid reads naturally next to Spark results
• Query history records the engine: appendQuery(connectionId, sql, engine) stores engine?: 'spark' | 'sql' on each entry. Backward-compatible: missing field treated as 'spark'
• DataFrame Viewer plumbing: new dataFrameViewerProvider.updateTotalRows(uri, total) for async count delivery; getEngine(uri) / setEngine(engine, uri) / refreshEngineContext() for per-tab engine state; showLivyResult / showResult gain engine and countPending parameters threaded through the cached CachedResult.grid, the initial HTML render, and the _sendUpdateData / _sendDataToWebview postMessage payloads. Webview JS updates the row count footer dynamically via a new updateRowCountText message
• Spark session no longer required to open a new SQL editor: lakehouseExplorer.newSparkSqlQuery (Ctrl+N) lost its isSparkActive() guard. The tab opens regardless of engine readiness; F5 prompts for a Spark session only when actually targeted at Spark. Lets users compose T-SQL queries without a Spark session running
• Language display renamed "Spark SQL" → "SQL" in the VS Code language picker (id stays sparksql internally so all keybindings, completion providers, and editorLangId == sparksql when clauses keep working). Reflects that .sparksql files can run T-SQL too
• New SQL editor seed comment points at the new engine toggle: -- Write plain SQL and press F5 to run. -- You can also run T-SQL against the SQL endpoint (read-only) by switching language in left lower corner …
• Files touched: src/sqlEndpointExecutor.ts (new, ~370 LOC), src/models.ts, src/authProvider.ts, src/queryHistory.ts, src/connectionManager.ts, src/fabricApiClient.ts, src/sparkSqlCodeLensProvider.ts, src/dataFrameViewerProvider.ts, src/extension.ts, package.json, CLAUDE.md. Bundle size grew ~1 MB from tedious (pure JS, no native binaries, bundles cleanly with esbuild)

1.9.3

• Generate Script... submenu on tables: right-click any table in the Lakehouse Explorer → Generate script... → INSERT INTO / UPDATE / MERGE INTO. Opens a new .sparksql tab prefilled with the right syntax, leading-comma column lists, NULL placeholders, and TODO markers (e.g. WHERE 1 = 0 guard on UPDATE; --TODO: real join key on MERGE). MERGE uses Delta's UPDATE SET * / INSERT * shortcuts so wide tables stay readable. Audit columns are excluded via the new lakehouseStudio.scriptGenerationExcludeColumns setting (default ["ewhTimestamp"], case-insensitive). Columns come from the in-memory sqlSchemaRegistry (loaded at session start), so generation is instant. Sits directly under "SELECT top 1000 rows" in the right-click menu — that command stays standalone
• Local Spark mode removed: Livy is now the only backend. Two architectural mitigations on the Fabric side closed the gaps Local Spark was solving — Fabric custom live pools (min-instances=1) collapse session startup to seconds, and Single Node environments cut per-session capacity so an F4 hosts 3+ concurrent sessions. Deleted: src/localSparkBackend.ts, python/local_spark_driver.py, bundled jars/ (~150 MB), hadoop/bin/winutils.exe + hadoop.dll, tools/fetch-jars.ps1. Settings lakehouseStudio.localSparkPythonPath and lakehouseStudio.localSparkDriverMemory removed. Java/JDK is no longer required locally; Python 3.12 is still required (used by the notebookutils shim executed in Livy sessions)
• notebookutils shim trimmed: python/notebookutils_shim.py keeps only the Livy path. Removed _Credentials (Key Vault REST), _Fs (Hadoop py4j), _RuntimeContext / _Runtime classes, the install(mode='local') branch that registered a fake notebookutils module in sys.modules, plus the _http_request_with_retry / _redact / urllib helpers. install() now takes no args and only monkey-patches notebookutils.notebook.run / runMultiple / exit onto Fabric's real module. notebook.run and runMultiple are preserved end-to-end (the v1.9.1 port stays in place)
• TS dispatch un-branched: dropped session.backend === 'local' checks across executeStatement, endSession, suspendCurrentSession, _waitForSessionReady, _startSessionPoll, updateStatusBar. _createLocalSession() deleted. Env picker no longer shows the Local Spark entry; Service Principal gate removed. getActiveBackend() removed (callers updated). Status bar always reads "Fabric Spark"
• Display path collapsed: __ECIT_LOCAL_PARQUET__ marker handlers removed from src/extension.ts (handleLocalParquetMarker, parquet wrapper in executeSelectViaJson, Preview Table parquet branch) and from src/fabpyNotebookController.ts (inline notebook cell handler). Three markers down to two (__ECIT_DF__ JSON + __ECIT_PARQUET__ OneLake). "All Rows" toggle is always visible
• Delete Table simplified: always runs spark.sql("DROP TABLE IF EXISTS …"). The notebookutils.fs.rm(abfss_path, recurse=True) branch that handled OSS Delta 4.0's metadata-only DROP is gone — Fabric's metastore DROP wipes the folder too
• DataFrame Viewer improvements preserved: header-fit column widths (Math.max(c.length * 9 + 40, 100)), Tabulator fitDataFill layout, typed CachedResult union, per-tab Tabulator caching, fast tab switching via setColumns + replaceData. All untouched — they're backend-agnostic
• Code shrink: ~1500 lines of TS + Python deleted, ~150 MB of bundled JARs/winutils dropped from the .vsix. The SQL rewriter, class-level monkey-patches (SparkSession.sql, DataFrameReader.json RDD overload, DataFrameWriter.saveAsTable, Catalog.tableExists, TimestampType.fromInternal, createDataFrame parquet roundtrip), Windows-hostility shims, and JAR version management are all gone
• Click-to-cache table feature removed: the per-table cloud-download icon, green-when-cached state, and right-click "Uncache" command are gone. Originally added in v1.8.2 mainly as a Local Spark win (cache lived in local JVM); on Livy the cache lives in the Fabric cluster and is wiped on every session restart, making the UX marginal and the state-tracking a small recurring source of tree-provider bugs. Deleted: src/cacheManager.ts (253 lines), tree-provider cache rendering, three commands, three menu entries
• Python worker faulthandler on Livy: spark.python.worker.faulthandler.enabled=true now set at Livy session creation. Surfaces real Python tracebacks when a worker crashes instead of an opaque Java EOFException. Diagnostic only, no perf cost. Other Local Spark perf configs (broadcast threshold, partition sizes, AQE knobs) deliberately not ported — they were laptop-tuned and risk OOM on small Fabric executors

Historical: If a local kernel is ever needed again, the full implementation (driver, shim local-mode, class-level monkey-patches, Windows fixes) is recoverable via git log --grep "Local Spark" — see v1.9.0 (8edc157) and v1.9.1 (100e5e8). Click-to-cache: git log --grep "cache".

1.9.1

• notebookutils.notebook.run / runMultiple ported into the Python shim: replaces ~600 lines of brittle TS regex orchestration (multi-line normalization, signature capture, marker-then-DAG two-statement flow, _buildRunMultipleCode Python-code generator) with real Python functions in python/notebookutils_shim.py. Customer code's own if/for flow drives execution — no more regex-parsing customer cells. Same implementation runs on both backends: Local Spark via sys.modules['notebookutils'], Livy via surgical monkey-patch on Fabric's real notebookutils.notebook (install(mode='local'|'livy')). notebook.exit(value) now returns the value to the caller via _NotebookExitSignal(BaseException) so except Exception: in the target can't accidentally swallow it. Per-thread recursion guard prevents a notebook running itself transitively
• Per-cell notebook registry preload: before submitting any customer cell, TS does a cheap includes() short-circuit, then (if matched) runs a discovery-only regex to collect notebook.run("name") literal names, walks the transitive closure via WorkspaceNotebookResolver, and prepends _ecit_notebook_registry.update({...}) to the cell. Customer's git checkout is the single source of truth for notebook content on both backends — no Fabric REST involved. File edits picked up on next cell run, no session restart needed. Dynamic names (notebook.run(f"nb_{x}")) surface a clear FileNotFoundError with a manual-preload hint
• Cell-boundary progress reset: TS prepends _ecit_reset_progress() to any cell with notebook.run / runMultiple so the progress grid shows only the current cell's activities. Multiple sequential notebook.run calls in one cell still accumulate into one grid (reset is per-cell, not per-call) — fixes leakage from previous cells
• Builtins shadowing fix in shim: on Livy the shim source is exec'd directly into the session namespace, which is polluted by from pyspark.sql.functions import * shadowing max, round, etc. with column-functions of the same name. Calling max(0, x) raised TypeError: max() takes 1 positional argument but 2 were given in _pop_depth cleanup. Fix: capture _max, _min, _round, _sum aliases at module top, immune to downstream import *. Local Spark dodged it via its own module namespace; Livy now safe too
• TimestampType Windows-safe deserialization (Local Spark): monkey-patched TimestampType.fromInternal and TimestampNTZType.fromInternal to fall back to datetime(1970,1,1) + timedelta(microseconds=ts) when datetime.fromtimestamp(ts) raises OSError [Errno 22]. Windows' C runtime _localtime64 rejects timestamps outside roughly years 1970–3000; BC's 9999-12-31 "end of time" sentinels were crashing .collect() on Local Spark. Linux/Livy unaffected by the underlying issue
• Python worker faulthandler re-enabled: re-added spark.python.worker.faulthandler.enabled=true and spark.sql.execution.pyspark.udf.faulthandler.enabled=true to Local Spark. They surfaced the actual TimestampType OSError when a Python worker crashed mid-.count(). Cheap to keep, occasionally diagnostic
• TS controller simplification: deleted _normalizeMultilineNotebookRun, the notebook.run / runMultiple signature-capture regex paths, the two-statement marker-then-DAG flow, and _buildRunMultipleCode. _executeViaLivy is now a single-statement send: preload → executeStatement → render. fabpyNotebookController.ts shrinks ~1378 → ~997 lines
• %run_multiple magic → shim call: TS now translates %run_multiple nb_a, nb_b (list form) and %run_multiple { "activities": [...] } (DAG-JSON form) into notebookutils.notebook.runMultiple([...]) calls so both flow through the shared shim path

1.9.0

• Local Spark — full customer-notebook compatibility: every common "Python data → Spark DataFrame" entry point now routes around the Windows + Spark 4.0 Python-worker crash by materializing through a local file. Patched in _install_spark_patches (class-level monkey-patches in python/local_spark_driver.py): spark.createDataFrame(pandas_df), createDataFrame([list_of_dicts], schema), createDataFrame([], schema) (synthesized via spark.range(0).selectExpr(...), no Python workers), and the RDD overload of spark.read.json(rdd) (collect → write JSONL → read from path). Customer notebooks across BC API extraction, SharePoint Excel ingest, skey notebooks, and gold dim/fact transforms now run unchanged on Local Spark
• Local Spark — Catalog.tableExists patch: spark.catalog.tableExists("schema.table") now checks for _delta_log/ at the ABFSS path (via Hadoop FileSystem.exists over py4j) instead of consulting the empty local catalog. Fixes silent data loss where add_skeys always started from sk=1 and write_df_to_lakehouse_table(..., load_type='append') silently fell into its overwrite branch
• Local Spark — env var forwarding: connection.environmentVariables (notably KEY_VAULT_NAME) is now passed in the init envelope and applied to the driver's os.environ. Customer get_secret(...) resolves to the right Key Vault locally, matching Livy behaviour
• Local Spark — local-disk parquet display path: .sparksql fast-mode + display(df) in notebook cells + Preview Table (Ctrl+3) now emit __ECIT_LOCAL_PARQUET__: instead of JSON-over-stdout on Local Spark. Driver writes a 500-row parquet file to a session-scoped tmp dir, Node reads it with hyparquet (same decoder used by Livy "All Rows"). Drops per-row JSON serialization for the visible result-display path; tmp dir is recursively deleted by LocalSparkBackend.stop() on session end / connection switch / VS Code reload (load-bearing cleanup; Python atexit is a defensive fallback). All Rows toggle is hidden on Local Spark (500-row cap is fixed)
• DataFrame Viewer — header-fit column widths + fitDataFill layout: column widths now derived from header text length (Math.max(c.length * 9 + 40, 100)) at all three column builders; Tabulator layout switched from fitData to fitDataFill. Skips Tabulator's per-cell text measurement (the slow path that took 5-10s on 500 rows × 40+ cols with fitData) — wide tables now render in well under a second. Applies to both Livy and Local Spark since they share the single viewer
• Delete Table — backend-aware: on Local Spark, Delete Table now runs notebookutils.fs.rm(abfss_path, recurse=True) instead of spark.sql("DROP TABLE …"). OSS Delta 4.0's path-based DROP only removes the catalog entry and leaves orphaned parquet + _delta_log/ behind; folder-delete actually wipes the table. Confirmation modal mentions which mechanism will run. Inactive-connection warning now names the connection so the user knows which one to switch to. Livy backend unchanged
• Class rename: LivySessionManager → SparkSessionManager, file src/livySessionManager.ts → src/sparkSessionManager.ts, helpers getLivySessionManager() → getSparkSessionManager() and isLivyActive() → isSparkActive(). The class manages both backends; the old name was misleading
• notebookutils_shim resilience: _get_token() and getSecret() now retry on transient failures (3 attempts, 1s/2s/4s backoff, only URLError / HTTP 5xx — 4xx surfaces immediately). Bearer <token> substrings redacted in any error body that leaks out
• CacheManager polish: _failAll(message) dedupes per-table change events via a Set<string>; failure error stored on each entry is now prefixed with the op kind ("cache failed: <reason>"). "No active session manager." replaced with the actionable "No Spark session available. Click the Spark status bar to start one."
• Status bar tooltips backend-aware: "Fabric Spark session …" vs "Local Spark session …" depending on the active backend
• Driver config cleanup: dropped both spark.python.worker.faulthandler.* configs (output never reached our terminal; never helped diagnose anything). Documented optimizeWrite=false / autoCompact=false as defensive-not-load-bearing. Driver _install_spark_patches collapsed _wrap_single_path + _wrap_parquet_reader into one parameterised helper. SQL rewriter docstring documents the known string-literal limitation
• Documentation: prerequisites list pyarrow + pandas as required for Local Spark (used by the createDataFrame parquet roundtrip). Patch-layer convention documented at the top of local_spark_driver.py so future layers land in the right place

1.8.2

• Click-to-cache tables in the Lakehouse Explorer: a cloud-download icon next to each table runs CACHE TABLE schema.table in the background; the table icon turns green when cached. Re-clicking a cached table runs REFRESH TABLE. Right-click → "Uncache" to free memory. Multiple clicks queue (FIFO, one in-flight) — Spark status bar spins while the queue drains. Cache state is per-session; ending the Spark session or switching connections clears it. Works for both Livy and Local Spark. Implementation: src/cacheManager.ts mirrors the notebookQueueManager pattern, dispatches via executeStatement, fires onDidChangeCacheState events that the tree provider listens to. Also adds local-Spark tuning configs aligned with the SPIP-Feather proposal: spark.sql.files.maxPartitionBytes=256MB (small dim tables in one partition) and spark.sql.execution.arrow.pyspark.enabled=true (Arrow optimization on result-marshaling paths)

1.8.1

• Local Spark: notebookutils shim: a subset of Fabric's notebookutils API is now re-implemented in python/notebookutils_shim.py and installed as a real module in sys.modules['notebookutils'] before user code runs. Customer code's import notebookutils / bare notebookutils.xxx resolves to our local re-implementation transparently — no if IS_LOCAL branches needed. Supported methods (same signatures as Fabric): credentials.getSecret(vault_url, name) (AAD OAuth2 → Key Vault REST), fs.rm(path, recurse) (Hadoop FileSystem.delete via py4j), runtime.context dict (currentWorkspaceName/Id, currentLakehouseName/Id), notebook.exit(value) (no-op print). See README "Supported notebookutils surface" for the full table
• Local Spark: DataFrame API write/read rewriting: df.write.saveAsTable("schema.table"), spark.table("schema.table"), and spark.read.table("schema.table") are now monkey-patched in the local driver so 2-part names are rewritten to delta.\abfss://@onelake.dfs.fabric.microsoft.com//Tables//`` against the active connection. Mirrors the existing SQL rewriter at the DataFrame API boundary so customer utility modules (e.g. dataplatform_functions.py's write_df_to_lakehouse_table) work unchanged in local mode
  Local Spark picker: Local Spark entry moved to the bottom of the environment picker (was top). Icon changed to $(device-desktop) (PC monitor) to make local-vs-Fabric distinction obvious

  1.8.0

  • Local Spark mode (opt-in, niche): optional PySpark 4.0 execution path for .sparksql iteration. Pick Local Spark at the bottom of the environment picker. Useful for saving capacity CU during heavy iteration or working around session quotas — Livy remains the default and is faster for one-shot queries against uncached data. Requires Service Principal auth, an active Fabric capacity, Python 3.12, PySpark 4.0 (pip install 'pyspark>=4.0.0,<4.1'), and Java 17+ (21 recommended). See README "Local Spark Setup" for the full guidance on when to use it vs. Livy. Bundled Scala-2.13 JARs live in jars/, refresh via tools/fetch-jars.ps1. Settings: lakehouseStudio.localSparkPythonPath (default python), lakehouseStudio.localSparkDriverMemory (default 32g)
  • SQL execution path: CACHE, UNCACHE, REFRESH, TRUNCATE, USE, ANALYZE statements are now treated as non-display statements (previously silently dropped → "0 statements executed"). Affects both Livy and Local Spark

  1.7.9

  • Folder-driven auto-import for Spark sessions: leave lakehouseStudio.autoLoadModules empty and the extension now auto-loads every top-level .py file from the configured utility folder (default utility/) at session start, sorted alphabetically. A populated comma-separated list still narrows to those modules. Same resolution drives scratchpad import lines so renaming or adding a utility no longer needs a settings update
  • Drop the nb_ prefix requirement for utility module imports: from <name> import * (cell-level, nested in .Notebook, and nested in utility .py) now matches any module name. Stdlib/3rd-party imports like from typing import * pass through untouched. Renamed modules without the legacy nb_ prefix work everywhere
  • Fix SyntaxError: import * only allowed at module level when calling notebookutils.notebook.run("X") against a target notebook with renamed (non-nb_) utility imports. The DAG builder now widens its hoist + strip regex to match any from <name> import * and pre-resolves utilities across all activities (was only the first), so utility imports never leak into the def _run_<activity>(): wrapper

  1.7.8

  • Default Lakehouse menu: right-click any notebook → Notebook metadata → Default Lakehouse... to pick a lakehouse and write default_lakehouse + default_lakehouse_workspace_id into notebook-content.py metadata. The picker lists lakehouses from the Dev-environment connection's lakehouse workspace (falls back to the active connection). Quick-pick shows <Lakehouse Name> with the workspace name as subtle context
  • Notebook tree warning: notebooks whose default_lakehouse GUID isn't in any configured connection now show a yellow $(warning) icon and tooltip — catches notebooks copied from another customer's workspace before they break execution. Cached by file mtime so refresh stays cheap
  • DataFrame Viewer instant tab return: each .sparksql tab now keeps its own Tabulator instance inside the webview (tablesByUri map, max 10, LRU-evicted). Switching back to a previously-loaded tab is just a CSS show/hide instead of destroy + recreate Tabulator — wide tables (50+ columns) that previously took ~700 ms now switch in <50 ms. Initial query rendering unchanged
  • Auto-build intellisense on first connection save: new connections trigger Refresh All Columns (Full Rebuild) automatically so the modal no longer asks the user to do it manually
  • Rename Rebuild Intellisense → Refresh All Columns (Full Rebuild) to clarify what it does compared to the background incremental sync
  • Trailing blank line on notebook save: serializer now ends notebook-content.py with a final blank line, matching what Fabric's git integration writes — no more phantom one-line diff after every commit
  • Spark status bar: green text is now applied immediately on activation (previously the orange placeholder lingered until the first state change). Idle/busy use the original brighter green tones for better visibility on light theme

  1.7.7

  • ▶ Run CodeLens above each statement in .sparksql files — click to execute just that statement without positioning the cursor. F5 / Ctrl+Enter still run the whole file (or selection)
  • Subtle blue separator line under each terminating ; in .sparksql files for visual multi-statement rhythm
  • Double-click a cell in DataFrame Viewer → copies the value with a brief green flash. Right-click menu (Copy Cell / Row / Column) unchanged
  • New CSV export button in DataFrame Viewer toolbar, next to Excel. UTF-8 BOM + RFC-correct escaping for commas, quotes, and newlines
  • Query history: $(history) icon in DataFrame Viewer title bar opens a QuickPick with the last 10 queries for the active connection. Each entry shows timestamp, the engine it ran on (Spark SQL / T-SQL endpoint) and the first 80 chars of SQL (searchable via matchOnDetail); selecting opens the query in a new untitled .sparksql tab. Persisted per-connection at local_development/connections/{id}/query_history.json
  • NULL now renders as italic brand-grey "NULL" in the data grid (cleaner than the previous purple tint). Light theme keeps the yellow cell background
  • Rename "Show all rows (Parquet)" → "Show all rows"
  • Remove first-query warm-up hint from the Executing screen — just "Executing..." now

  1.7.6

  • Reduce Livy session keepalive default from 480 min (8 h) to 60 min (1 h) — most sessions don't need to stay warm for a full workday. Configurable via livySessionKeepAlive setting
  • Redesign DataFrame Viewer loading UI: the 10×12 pulsing skeleton grid is replaced with a compact rotating arc spinner (primary-blue, 32 px) above an "Executing..." label. Post-execution "Loading data..." now shows a full green bar instead of the 3-dot pulser, signalling completion
  • Rename Lakehouse Explorer commands for clarity: "New Spark SQL query" → "SQL editor" (Ctrl+N), "New notebook scratchpad" → "Draft a notebook"

  1.7.5

  • Fix DROP TABLE in .sparksql files: DROP was missing from the non-display statement regex, causing DROP TABLE commands to be silently ignored
  • Fix "first time can be slow" warm-up message appearing on every query instead of only the first: flag is now consumed in showPreparing() (covers all outcomes) and reset on new session start
  • Escape triple-quotes and backslashes in SQL before wrapping in spark.sql("""...""") for single-statement execution paths (multi-select and notebook controller already escaped correctly)

  1.7.4

  • Session keepalive heartbeat: sends SELECT 1 every 10 minutes to prevent Fabric from timing out idle Spark sessions. Controlled by new livySessionKeepAlive setting (default 480 min / 8 hours, set to 0 to disable)
  • Also passes heartbeatTimeoutInSecond to the Livy API at session creation as an experimental server-side idle timeout

  1.7.3

  • Subtle executing query text: timer, pipe, and message now match the "Showing X rows" muted style instead of bright secondary text
  • Enhanced column refresh on Ctrl+Shift+R: detects schema.table references in the active editor (.sparksql files and Python spark.sql() blocks) and refreshes their columns from OneLake — no more stale columns after table schema changes

  1.7.2

  • Brighter "Executing query..." text in DataFrame Viewer: switched from faded hint color to secondary text color for better visibility
  • Preview Table (Ctrl+3) now supports backtick-quoted table names (e.g. schema.\some_table``) — common for Spark tables with special characters

  1.7.1

  • Fix multi-line notebookutils.notebook.run(...) hijacking: calls spanning multiple lines (e.g. with keyword args like timeout_seconds, arguments={}) are now normalized and intercepted correctly
  • Fix keyword args support in notebook.run parameter passing: handles both positional (name, timeout, {params}) and keyword (name, arguments={params}) signatures
  • Fix Shift+Enter (executeCodeInCell) now delegates to _executeViaLivy, giving it full preprocessing (inlining, hijacking, %pip transform) instead of raw execution
  • Fix notebook auto-discovery for nested folder structures: findFirstNotebookRoot now finds the common parent when notebooks are spread across sibling subfolders (e.g. nb_silver/, nb_gold/)
  • Fix multiple from workspace.xxx import * in the same cell: all imports are now inlined, not just the first
  • Fix progress grid duration: shows each notebook's individual run time instead of cumulative elapsed time

  1.7.0

  • display(df) support: injected at Livy session startup, routes DataFrame results to the DataFrame Viewer panel (works in both Ctrl+Enter and Shift+Enter)
  • Auto-load modules on session start: new autoLoadModules setting loads configured modules (e.g. nb_dataplatform_functions) automatically when a Spark session starts, and pre-fills them in new scratchpad notebooks
  • Simplified "Open with Read Code": parquet/CSV/Excel snippets now use simple relative Files/ paths instead of FILES_BASE_PATH with IS_FABRIC conditionals
  • Removed scratchpadImports setting (replaced by autoLoadModules) and notebookWarnings setting
  • Settings reorganized: utilityModulesPath and autoLoadModules grouped together for discoverability

  1.6.3

  • Redesigned Add/Edit Connection form: authentication moved to top, workspace and lakehouse pickers via Fabric API ("Select..." buttons) replace manual GUID entry
  • Connection name is now optional ("Connection Friendly Name") — defaults to the lakehouse display name from Fabric API
  • Unified Add and Edit connection forms into a single codebase, eliminating ~400 lines of duplication
  • Renamed right-click menu "Refresh intellisense" to "Rebuild Intellisense"
  • Post-save modal dialog guides new users on starting a Spark session and rebuilding intellisense

  1.6.2

  • Feature demo GIFs added for all major features (Livy sessions, Lakehouse Explorer, notebooks, Spark SQL, DataFrame Viewer, intellisense, Schema Compare, capacity management)
  • README overhaul: added internal tooling disclaimer, authentication requirements, recommended VS Code settings, and login troubleshooting tip
  • Removed old Setup-guide.md and screenshot attachments — all setup info now lives in README
  • Included docs folder in .vsix package so GIFs render on the marketplace

  License

  MIT