Skip to content
| Marketplace
Sign in
Visual Studio Code>Other>Cluster WatcherNew to Visual Studio Code? Get it now.
Cluster Watcher

Cluster Watcher

Cameron Braunstein

|
4 installs
| (0) | Free
Monitor Slurm jobs and capacity, manage the local service, and reopen SSH sessions from the VS Code sidebar.
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

Cluster Watcher for VS Code

This extension adds Cluster Status above My Jobs to the Activity Bar. It is a thin client for the loopback APIs provided by cluster-watcher serve; the Python application remains responsible for configuration, SSH, MFA, Slurm queries, and refresh scheduling.

When no backend service is found, My Jobs and Cluster Status become collapsed offline placeholders. Cluster Status retains its service switch, which is labeled Start Server while off. Run setup wizard remains in the Command Palette. A thin gray line separates the cluster cards from the service toggle and dashboard URL, followed by view sessions... and configure clusters..., which opens the configuration file; configuration and settings commands remain in the Command Palette. Successful backend responses restore the live views. You can still expand an offline placeholder manually.

Requirements

  • On first use, the extension looks for an existing cluster-watcher command on the machine running the extension (the remote host under Remote SSH). If it cannot run one, choose Install and Add to PATH, Use Only in VS Code, or Choose Existing Executable. The first two choices download the exact backend version paired with the extension and verify its embedded SHA-256 checksum before running it. You can change the choice at any time with Cluster Watcher: Manage Backend Installation. In an unpackaged Linux or macOS development checkout with no backend-manifest.json, the first choices are Download Latest Release and Add to PATH and Download Latest Release (No PATH Change). They run the checkout's install.sh, verify the release against SHA256SUMS, and select the installed command automatically. Build from Source is available in the same menu if the release binary cannot run; it requires Python 3.11+ and build dependencies. A development download may be older than the extension checkout. No-PATH choices leave shell profiles unchanged; add-to-PATH choices use the installer's safety checks. Development-checkout installs are standalone external commands, so remove one with the checkout's uninstall.sh, not Uninstall Managed Backend.

  • A manual installation remains supported. On Linux or macOS, install the prebuilt executable with

    curl -fsSL https://raw.githubusercontent.com/CameronBraunstein/cluster-watcher/master/install.sh | bash -s -- --add-to-path
    

    (or pipx install git+https://github.com/CameronBraunstein/cluster-watcher), then run cluster-watcher setup once to describe your clusters. On native Windows, install it with the repository's install.ps1; SSH key/agent clusters are supported there. Password/OTP clusters require the complete Unix OpenSSH multiplexing path, so run both the executable and this workspace extension inside WSL for those clusters.

  • SSH access to each Slurm cluster from that machine.

  • If an existing executable is not on PATH, select it through Manage Backend Installation. clusterWatcher.executable remains a fallback for settings created before the managed installer was introduced. On Linux, the prebuilt installer verifies glibc through Node's runtime report or getconf GNU_LIBC_VERSION. If the prebuilt backend is incompatible or its glibc version cannot be verified, run ./install.sh --from-source --add-to-path in a matching source checkout, then choose the resulting command with Choose Existing Executable. Use its absolute path if the extension host has not picked up the new PATH.

My Jobs uses collapsible state groups instead of terminal-style banners. On first startup, the service can be running before its first Slurm collection finishes. During that interval, offline My Jobs, Cluster Status, and status-bar views show Loading rather than start-service controls. The extension retries the backend's temporary first-collection response every second; each view replaces Loading as soon as its own data arrives. Each job is another collapsible card whose compact form contains its name, progress color bar, and completion/start estimate. Long names stay on one line and end in an ellipsis; hovering the title (not the rest of the card) shows an immediate, white-outlined label with the cluster, job ID, and full job name. Card summaries use tight internal padding and reduced spacing between neighboring cards. The copyable job-ID badge has rectangular corners and an inset blue hover/focus border whose four edges remain visible. Dependency-blocked cards omit the empty progress bar and show dependency:<type> <job ID>; the ID links to the relevant card and a popup reports when it cannot be found. In the expanded Depends on row, a clock after an ID marks it as waiting, a green check marks it as satisfied, and a red x marks it as failed. Hovering an icon shows the Slurm status and a fuller description instead of displaying a status parenthetical, using the same fast hover timing as the availability breakdown. Because Slurm normally removes satisfied IDs from its remaining-dependencies field, checks appear only while the ID remains available in the expression or a retained snapshot. Jobs whose overall condition is impossible appear under Failed Dependency instead of Pending. Open cards use a horizontal divider between the always-visible summary and their details instead of a disclosure icon. Opening or closing highlights the card border immediately, then fades that highlight more slowly. Pending cards with usable estimates use <duration> estimated wait; missing estimates show no estimate available without an empty bar. The bar and run/allotted time stay on one line at the sidebar's narrowest expanded width, including on completed and failed cards; their denominator comes from the time limit retained in Slurm accounting. Zero-length recorded durations appear as <1m rather than now. Jobs are separated into Failed (Early) and Failed (Timeout) groups; the latter is reserved for Slurm's explicit TIMEOUT state. Expanding a card reveals the job-ID badge to the right of its progress bar (or priority gauge); click it (or focus it and press Enter) to copy it without collapsing the card. The cluster, partition, and resource summary follow in the details. A running job's calculated limit and a terminal job's end time appear with the expanded lifecycle details. Elapsed time is not repeated there because it is already in the compact progress fraction. Running cards omit Ended; pending cards omit both Launched and Ended. Expanded resource metadata wraps across as many lines as needed. Lifecycle timestamps stay on one line when space permits; on a narrow card, the date remains on the first line and the time wraps onto a right-aligned second line. The compact icon/short-label action row remains on one line. Long progress labels ellipsize, while dependency expressions and expanded job names wrap inside the card. When a cluster exposes a global queue rank—or numeric Slurm priorities for at least two of the user's visible pending jobs—expanded Pending cards add a number-free low-to-high priority gauge. Hover it for Global priority <rank> / <all pending> and Your priority <rank> / <your pending> on separate lines, or hover the Priority label to the left of the bar for a reminder that lower rank numbers are better and a short scheduling caveat. The marker visualizes the global rank and falls back to the user rank only when global data is unavailable. If no wait estimate exists, the gauge appears in the collapsed summary and becomes as thick as a normal progress bar; opening the card moves it into the expanded details. It has no directional arrows and is omitted when no useful comparison is possible. Finished, failed, cancelled, and other non-running/non-pending cards show the archive-box icon on the compact card, just left of progress. Running and Pending cards keep Archive in their expanded actions; Archive cards keep Restore there. Expand a card to see details and to:

  • archive a running or pending job, or Restore a job from the collapsed Archive group at the bottom;
  • once a job has left the pending group, select .err or .out to open that stream as a read-only VS Code document. Pending cards omit both actions; an archived pending card receives refreshed state and gains them when it starts or finishes;
  • select script to view the Slurm batch script the job ran, as a read-only, shell-highlighted document. For queued and running jobs this is Slurm's exact copy of the submitted script. Slurm usually discards that copy when a job ends, so for finished jobs the extension opens the script file named on the recorded sbatch command line instead, and a warning says that it is the current file and may have changed since submission. The .err, .out, and script controls preserve the sidebar's current scroll position when their document opens;
  • for running and pending jobs, select End in the card's lower-right corner. A modal dialog asks for confirmation, then the service runs scancel for that job. The ID badge turns red and the button disappears until Slurm stops listing the job;
  • for jobs with a Slurm dependency, read the Depends on row. Each job ID in it is a link that opens and scrolls to that job's card (opening its group if collapsed). The status icon follows the ID only in this expanded row and provides its description on hover. A notification explains when the job is not in the list.

The archive is stored in VS Code extension global state and survives editor restarts. Group, job-card, and cluster disclosure choices survive live data refreshes: the webview reports each expand/collapse to the extension, which renders the remembered state into every refreshed page (for the current VS Code session). Archive and restore actions also retain the current viewport by anchoring it to a nearby unchanged card across the rerender. Cluster names are collapsible headings in Cluster Status. The native My Jobs and Cluster Status view headings replace redundant titles inside each webview. A compact Last update: <date and time with seconds> line appears directly below each heading. Each heading has its own Refresh action, which polls only that view's endpoint. Cluster Watcher: Refresh Sidebar in the Command Palette refreshes both views; service, dashboard, configuration, settings, and backend-management actions also remain available there. A pressed heading refresh button changes briefly to a spinning sync icon, including when the response is unchanged. A notification distinguishes new data (Refreshed at:) from an unchanged check (Checked, but no updates at:), with a local date and time including seconds.

Log documents initially contain the newest 2,000 lines. If more output exists, use the upward-arrow Load 2,000 Older Lines editor-title action; each click prepends one more bounded page until the complete remote file is loaded.

When the service is not reachable, the collapsed jobs and cluster placeholders contain no duplicate startup buttons. Run setup wizard runs cluster-watcher setup in a terminal and remains available in the Command Palette even when a TOML already exists. Edit configuration opens the file reported by cluster-watcher config --path in an editor tab; if missing, it offers the wizard. Saving the configuration from VS Code validates it immediately (via cluster-watcher list) and reports any error; restart the service to apply it.

The extension-managed backend is installed per user on the extension host:

  • Linux: ~/.local/share/cluster-watcher/vscode-backend/bin
  • macOS: ~/Library/Application Support/Cluster Watcher/vscode-backend/bin
  • Windows: %LOCALAPPDATA%\Programs\ClusterWatcher\vscode-backend\bin

Use Only in VS Code exposes that directory to new integrated terminals through VS Code's terminal environment API without changing the system PATH. Install and Add to PATH additionally writes one marked block to the current Unix shell profile, or one entry to the Windows user PATH. Open a new terminal after changing PATH. The ownership manifest records the exact change; Manage Backend Installation can add or remove it later without touching a pre-existing PATH entry.

The same management command can switch between the managed and an existing executable, repair/update the managed copy, or uninstall it. Switching to an external executable keeps the managed copy available for a later switch. Uninstalling the managed backend removes only extension-owned files and PATH changes and preserves Cluster Watcher configuration. Removing the VS Code extension itself intentionally leaves a standalone managed backend in place; use the management command first if it should also be removed.

If a service is reachable but was started without --jobs-api (for example by hand as cluster-watcher serve), Cluster Status still works, while My Jobs explains the problem with Copy restart command and Retry buttons, and the status bar shows jobs API off rather than offline. Start service warns instead of attaching to such a service. Stop it and start it again with --jobs-api. Buttons that an older service does not support yet ask you to upgrade cluster-watcher and restart the service.

Refreshes are conditional: the extension sends the previous response's ETag and the service answers 304 Not Modified when your jobs and the cluster status are unchanged, so the views are not rebuilt (no flicker, and scroll and hover are kept). Progress bars, elapsed times, and wait estimates still count on their own every few seconds. The compact freshness line below each heading shows only when data last changed, with exact second precision.

Pending-job priority bars show both a cluster-wide rank among all visible pending jobs and a user rank among your pending jobs on that cluster; 1 is the highest Slurm score and tied scores share a rank. The marker visualizes the user rank, while hover text reports both ranks. The ⓘ beside each bar begins Higher scores … and explains that neither value guarantees launch order or start time. A better rank usually means earlier consideration than lower-ranked jobs when their resource requirements and scheduling tier are comparable, but QOS, reservations, resource fit, and backfill still control actual launch order. If the cluster denies the queue-wide query, global priority is shown as unavailable while the user rank remains available. The former large value was Slurm's raw, cluster-specific weighted priority score. Its absolute magnitude is not a queue position and is not comparable between clusters, which is why the interface now presents relative global and user ranks.

Cluster Status shows each partition as a permanently expanded compact card. It shows the partition name, seven account-aware launchability blocks for 1, 2, 4, 8, 16, 32, and 64 GPUs, and the best GPU's compact model label (such as H100), VRAM, TFLOPS/s, CPU-thread total, and GPU availability. Each block uses the configured wait thresholds from dark green through green, yellow, orange, and red. Black means the request was rejected by account/QOS policy, gray means unavailable or unprobed, and a striped block exceeds the partition's capacity. Hover a block for its exact one-hour-job estimate, expected start, or Slurm error, and hover the GPU model for its full name. VRAM and thread hover text explicitly labels those amounts. When the partition name and complete launchability strip do not fit together, the strip wraps intact and flush left onto a second line. Thin gray lines separate clusters. Each partition has one FairShare track with a marker for every matching account association. Hover an individual marker to highlight it and see account_name: rank/total. Hover boxes fit their text and stay within the sidebar, wrapping long account names when needed. The background track has no hover tooltip. FairShare does not run startup dry runs or attempt to choose a submission account. Slurm uses the association for the account a job is submitted under, rather than averaging the markers. To inspect an actual pending job, use scontrol show job JOB_ID for its account/QOS and sprio -j JOB_ID -l for its priority components (with the multifactor priority plugin).

Free GPUs precedes a green/red bar with its idle/total fraction; hover the bar for a node-only state breakdown (zero-count categories are omitted).

The card does not repeat the same wait information as a graph. The colored blocks keep counting down locally between service updates. A development partition capped below one hour shows the actual shorter probe runtime in its hover text. The main README explains the scheduling semantics; unavailable data is simply omitted.

Cluster Status reports the service as starting, running, stopping, or stopped. Both a compatible service started elsewhere and a tmux service show only the clickable dashboard URL, with no status prefix. The URL is a button styled as a link, so its process/start tooltip is not accompanied by a dummy # target. A smaller toggle appears to the left of smaller text, preserving the standard local URL at minimum sidebar width; it is green when on and gray when off, and its hover says Stop service or Start service. When off, Start Server appears next to the toggle; there is no duplicate start button. A running tmux service shows the URL without the redundant “Running in the background” label. Hover the URL for the backend process ID and start time. Cluster headings show the cluster name and job-ratio bar without the login node. view sessions... opens the SSH Sessions popup, a selectable dialog in Cluster Status that spans nearly the full view width, with a single scrollbar and an × close button close to the upper-left edge. Entries show name: username@node followed by 🕒 HH:MM left on the next line (hours and minutes). Popup text uses the gray My Jobs update color at a slightly larger size, with a larger × close control. Cluster and login names share one line as name: username@node, without extra indentation. Long account names wrap within the login address instead of moving the whole address below the colon. Cluster Status partition cards and My Jobs cards have square corners. Black side borders and inner padding give the text breathing room. While the popup is open, the background scrollbar is hidden and the sole visible scrollbar scrolls sessions. Sessions follow the machine order in clusters.toml. Gray lines separate the popup heading and each session. The popup height fits its content and stays centered, with minimal space below the last session and a viewport height limit for scrolling long lists. Cluster headings and their gray dividers use compact spacing. Every logged-out interactive machine replaces the job ratio with Log in. A machine with credential_group = "site" also has Log into all machines on site; the first action targets only that machine, while the second reuses the shared password for all closed group members. Session inspection errors also appear in Cluster Status, so there is no separate Server Status view. Stopping an attached local service asks for confirmation, warns that other connected clients will be disconnected, and then requests a graceful shutdown from the backend. Below a thin gray divider, the service toggle and dashboard URL appear before configure clusters.... That link opens the selected clusters.toml without moving the Cluster Status scroll position (or offers setup if the file does not exist). The extension starts the service in a detached tmux session on its host, so closing or reloading VS Code does not stop it. tmux must be installed there. The OpenSSH masters may remain reusable until their own connection or idle limit ends. A host reboot ends both the tmux service and SSH sessions; a host policy that kills user processes at logout can also end them.

Two settings control background feedback:

  • clusterWatcher.notifications (all, failures, or off; default all) shows a notification, with Open .out/Open .err buttons, when a job that was running or pending completes, fails, or is cancelled. The first refresh after VS Code starts only records the current state, so old completions are not replayed; jobs ended with End Job are not reported.
  • clusterWatcher.statusBar (default true) shows N running · M pending in the status bar, or a warning-coloured offline item when the service is unreachable. Its hover uses the correct singular/plural job labels and shows the date and time of the latest successful jobs refresh, including seconds. Click it to reveal My Jobs.

clusterWatcher.dateFormat (default DD.MM.YYYY) sets how sidebar dates are shown, using the tokens YYYY, YY, MM and DD (for example YYYY-MM-DD). Times follow as 24-hour HH:mm in local time, and a change redraws the sidebar immediately.

Webview buttons do not use command links. Each one posts a message, and the extension only runs commands from a fixed allowlist (archive, restore, open logs/scripts/dashboard, end job, start/stop/refresh the service, setup, edit configuration, settings, backend management, and login).

Use Cluster Watcher: Start Service & SSH Sessions from the Command Palette or Cluster Status. The extension launches serve --jobs-api --skip-login in a detached tmux session, then opens closed SSH sessions with a separate login helper. Password and OTP requests use masked, focus-retaining VS Code input boxes; responses pass over the helper's private stdin and never appear in its arguments, environment, configuration, or retained diagnostic output. If a compatible service is already listening at clusterWatcher.backendUrl, the extension attaches without starting another process.

The tmux session name starts with cluster-watcher-. Inspect it with tmux list-sessions and tmux attach -t <session-name> on the extension host; detach using Ctrl+B, then D. Stop service shuts the backend down. If the tmux session exists but the service is unreachable, Start service offers Restart Stale Session. It terminates only the named session after confirmation, then starts the service again. A cancelled restart leaves the session intact; a service that recovers before the restart proceeds is attached instead. When the HTTP service is offline, Cluster Status omits the redundant SSH session fetch error.

If a cluster's SSH session closes later, its Cluster Status card says so and shows Log in again. The button runs cluster-watcher login <cluster> as a background process, without restarting the service. Other closed clusters that share the same password (credential_group) are logged in at the same time, so the password is asked for once, followed by one one-time code per cluster. No terminal opens, and the card recovers on the next refresh. Cluster Watcher: Log In Again to Closed SSH Sessions in the Command Palette does the same for every closed cluster.

The cluster-watcher executable is resolved on the machine running the VS Code extension host. Set clusterWatcher.configPath when the normal configuration location should not be used.

The extension validates every managed or selected executable with <executable> --help and <executable> capabilities before starting the background service. The capability check prevents an older executable from being started with unsupported terminal-free credential arguments. If it fails, use the notification's Manage Backend button to repair the managed copy, or run ./install.sh --from-source --add-to-path from the matching source checkout and reload the extension host.

The extension is declared as a workspace extension. Under VS Code Remote SSH, install it on the remote side so its process, the executable, and reusable SSH control sockets all live on that remote development host. Backend selection is independent for local, WSL, Remote SSH, and container extension hosts. Workspace Trust is required because the extension downloads/runs a native executable and opens SSH sessions.

The extension is licensed GPL-3.0-or-later (LICENSE). The original Activity Bar icon and the Marketplace icon (media/icon.png, rendered from media/icon-source.svg) show server boxes under a magnifying glass and are dedicated under CC0-1.0; media/LICENSE.txt is included with the packaged extension. Release notes are in CHANGELOG.md.

See the repository's main README.md for development, VSIX packaging, publishing, security, and SSH-session sharing details.

  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
  • Your Privacy Choices
  • Consumer Health Privacy
© 2026 Microsoft