NEXUS - Synapse Notebook Explorer for VS Code

NEXUS (Notebook EXecution Unified System for Synapse) - A VS Code extension for editing, executing, and managing Azure Synapse Spark notebooks with interactive DataFrame visualization and advanced productivity features.

Quick Start

flowchart LR
    A[Configure Workspace] --> B[Authenticate]
    B --> C[Browse Notebooks]
    C --> D[Open & Execute]
    D --> E[View Results]
    E --> F[Publish Changes]
    
    style A fill:#e1f5ff
    style D fill:#fff4e1
    style F fill:#e8f5e9

1. Configure Workspace

Run NEXUS: Configure Workspace and provide:

• Azure Subscription ID
• Resource Group name
• Synapse Workspace name
• Default Spark Pool
• Authentication method (Azure CLI or Interactive OAuth)

2. Authenticate

Click the sign-in icon (🔑) or run NEXUS: Authenticate with Azure:

• Azure CLI: Uses existing az login credentials
• Interactive OAuth: Browser-based Azure AD authentication

3. Browse & Execute

• Browse notebooks in the NEXUS tree view
• Open notebooks to edit locally
• Execute cells with persistent Spark sessions
• Use magic commands (%run, %%sql, %%pyspark, %%markdown)

4. Publish

• Right-click notebook → Show Publish Diff
• Review side-by-side comparison
• Publish to workspace or revert changes

Features

🔐 Authentication

• Azure CLI: Automatic credential reuse from az login
• Interactive OAuth: Browser-based login with automatic token refresh
• Token Management: On-demand refresh, no manual intervention

📓 Notebook Management

• List & Search: Browse all notebooks with instant search filtering
• Pagination Support: Handles 1000+ notebooks efficiently
• CRUD Operations: Create, open, edit, delete notebooks
• Folder Support: Navigate nested notebook folders
• Local Cache: Fast loading with automatic cache warm-up

▶️ Execution

• Persistent Sessions: Each notebook maintains its own Spark session
• Session Reuse: Sessions persist across executions and VS Code restarts
• Auto-Restart: Configuration changes automatically restart sessions
• Manual Control: Stop sessions anytime via toolbar button
• Language Support: Python (PySpark), Scala, Spark SQL, Markdown
• Progress Notifications: Real-time execution status in VS Code

📊 Interactive DataFrames

Enhanced display() function automatically available in every session:

# Display any Spark DataFrame interactively
df = spark.read.parquet("data.parquet")
display(df)

Features:

• Sorting: Click column headers (↑↓ indicators)
• Filtering: Click 🔍 icon to search/select values
• Cascading Filters: Filter options update based on filtered data
• Resizable Output: Drag handle to adjust table height (200px-2000px)
• Pagination: Automatic handling of large datasets
• SQL Integration: %%sql cells automatically use interactive tables

🪄 Magic Commands

%run - Execute Notebooks Inline

# Run notebooks from any folder (up to 10 levels deep)
%run Includes/DatabaseUtils
%run helpers/validation

# Use functions/variables from loaded notebooks
result = utility_function()

Features:

• ✅ Folder path support: %run analytics/Prep
• ✅ Nested execution (depth ≤ 10)
• ✅ Circular reference detection
• ✅ Cache-first loading (fast)
• ✅ Language detection per cell

%%pyspark - Force Python Execution

%%pyspark
df = spark.read.csv("data.csv")
display(df)

%%sql - Execute Spark SQL

%%sql
SELECT category, COUNT(*) as count
FROM products
GROUP BY category

Results render in interactive tables with sort/filter/pagination.

%%markdown - Render as Markdown

%%markdown
## Analysis Summary
- Revenue increased by **25%**
- Costs decreased by 10%

📝 Publishing Workflow

sequenceDiagram
    participant User
    participant VS Code
    participant Diff View
    participant Synapse

    User->>VS Code: Edit notebook locally
    User->>Diff View: Show Publish Diff
    Diff View->>Synapse: Fetch workspace version
    Diff View-->>User: Side-by-side comparison
    User->>Synapse: Publish changes
    Synapse-->>User: ✅ Published

1. Make local changes in VS Code
2. Right-click → Show Publish Diff
3. Review added/modified/removed cells
4. Publish: Push changes to Synapse
5. Revert: Restore workspace version
6. Edit: Modify cells in diff view

⚙️ Configuration

Workspace Level (shared defaults):

• Subscription, Resource Group, Workspace
• Default Spark Pool
• Authentication method

Per-Notebook (override defaults):

• Spark Pool selection
• Environment configuration
• Executor size (Small, Medium, Large, XLarge, XXLarge)
• Executor count (1-40)

Access via settings icon (⚙️) in notebook toolbar.

Commands

Session Management

stateDiagram-v2
    [*] --> NotStarted
    NotStarted --> Creating: First cell execution
    Creating --> Starting: Session created
    Starting --> Idle: Session ready
    Idle --> Busy: Execute code
    Busy --> Idle: Execution complete
    Idle --> Shutting_Down: Stop session / Change config
    Shutting_Down --> [*]
    
    Busy --> Error: Execution failed
    Error --> Idle: Retry
    Starting --> Error: Startup failed

Lifecycle:

• First Execution: Creates new session (30-60 seconds)
• Subsequent Executions: Reuses existing session (<5 seconds)
• Isolation: Each notebook has independent session
• Configuration Changes: Auto-restarts session with new settings
• Manual Stop: Click ⏹️ button in toolbar
• Auto-Recovery: Failed sessions automatically removed

Active Sessions View:

• Real-time list of running Spark sessions
• Shows notebook name, pool, state, executors
• Auto-refreshes every 30 seconds
• Right-click to stop individual sessions

Requirements

• VS Code: Version 1.80 or later
• Node.js: Version 18 or later (for development)
• Azure Synapse: Workspace with configured Spark pools
• Azure CLI: For Azure CLI authentication (az login)
• Permissions:
  • Synapse Contributor or Synapse Artifact Publisher (for notebook CRUD)
  • Synapse Compute Operator (for execution)

Troubleshooting

Authentication

• Azure CLI not found: Install Azure CLI from https://aka.ms/install-azure-cli
• Token expired: Run az login or re-authenticate via command palette
• Interactive auth fails: Check Azure AD redirect URI configuration

Notebooks

• Empty notebook list: Verify workspace name, then click Refresh (🔄)
• Cannot create notebook: Ensure Synapse Contributor permissions
• Notebook won't open: Check notebook exists and you have Read access

Execution

• "Session does not exist": Wait 2 seconds after session creation, or stop and restart session
• Slow first execution: Session startup takes 30-60 seconds (normal)
• Config changes ignored: Stop session manually before changing settings
• %run fails: Verify notebook exists in workspace and depth ≤ 10

Performance

• Slow notebook loading: Large workspaces (1000+ notebooks) take time to paginate
• Search not working: Click Refresh first, then search
• High memory usage: Stop unused sessions via Active Sessions view

Debug Logging

Open Debug Console (View → Debug Console) to see detailed logs:

• Session creation and verification
• Livy API requests/responses
• Cache operations
• Error details with HTTP status codes

Known Issues

• Session startup: 30-60 seconds on first cell execution (Spark limitation)
• Large notebooks: 100+ cells may have slower diff loading
• Interactive auth: Requires manual redirect URI configuration in Azure AD
• Executor metrics: Not yet available in Active Sessions view

Release Notes

Version 0.0.1 (October 2025)

Initial release with comprehensive Synapse notebook management:

Authentication:

• ✅ Azure CLI + Interactive OAuth
• ✅ Automatic token refresh

Notebook Management:

• ✅ CRUD operations with pagination (1000+ notebooks)
• ✅ Search and filter
• ✅ Folder support
• ✅ Local caching

Execution:

• ✅ Persistent Spark sessions with reuse
• ✅ Per-notebook configuration
• ✅ Active Sessions view with auto-refresh
• ✅ Language support (Python, Scala, SQL, Markdown)

Advanced Features:

• ✅ Interactive DataFrame display() with sort/filter/pagination
• ✅ Magic commands (%run, %%sql, %%pyspark, %%markdown)
• ✅ Nested %run execution (depth ≤ 10)
• ✅ Publish/diff/revert workflow

Feedback

Report issues or request features in the GitHub repository. Contributions welcome!

See DEVELOPMENT.md for developer documentation.

Enjoy managing your Synapse notebooks! 🚀