PrairieLearn Review

A VS Code extension for reviewing questions in a PrairieLearn course repository. It adds a PrairieLearn Review activity bar container with a review queue, a question browser, and a tag editor, and shows the real PrairieLearn question preview in an editor tab. A loopback gateway proxies PrairieLearn and exposes the same review state to coding agents over MCP.

It does not fill answers, submit grades, edit question content, or run Git commands. The only files it writes are the tags arrays in questions/<qid>/info.json.

Requirements

• VS Code 1.100 or newer
• Node.js 22 or newer and pnpm (to build)
• Optional: the official GitHub Pull Requests extension and a GitHub.com sign-in for PR comment indicators
• For managed mode: Docker with a local prairielearn/prairielearn:us-prod-live image, unless an image pull is explicitly allowed
• macOS or Linux for supported managed-runner process cleanup

Install

pnpm install
pnpm package          # builds dist/ and writes prairielearn-reviewer-<version>.vsix
code --install-extension prairielearn-reviewer-*.vsix

For development, open this repository in VS Code and press F5 (Run Extension (fixture course)). This launches an Extension Development Host on tests/fixtures/course with the bundle rebuilding on save.

Using it

Open a folder that contains infoCourse.json (or set plreview.courseRoot). Then run PrairieLearn Review: Start from the Command Palette or from the welcome view in the activity bar.

• Queue: the ordered review queue. Click an entry to open it. Inline actions mark done, skip, re-queue, or remove. Drag entries to reorder them. Title actions step to the previous/next reviewable entry, start a new review with a fresh queue, add questions, and clear completed entries. The badge counts questions left to review. New Review… replaces the existing queue and opens the first selected question. An amber corner marker identifies branch changes (hollow) or changes since the question was last marked done (filled); a blue marker identifies resolved-only (hollow) or unresolved (filled) PR review threads.
• Questions: every assessment (in declared order) plus All questions. Use the inline + to enqueue a question or assessment. The 🔍 title action (Cmd+Alt+P / Ctrl+Alt+P) opens a quick search over QIDs, titles, tags, change state, and PR-thread state.
• Tags: the course tag catalog as checkboxes for the current question. Edits stay a draft (● unsaved) until you choose Save or Save Tags and Reload Course; Discard reverts them. Tags missing from the catalog are listed separately and preserved on save. If you navigate away with unsaved edits, a dialog asks whether to save or discard them. If an agent navigates instead, the draft stays with its original QID.
• Preview tab: the PrairieLearn question preview, served through the gateway. A status strip shows the active branch/PR, saved-review changes, and resolved/unresolved review threads. Its toolbar has previous/next, done/skip, new variant, reload course, and shortcuts to question.html, info.json, and server.py. Its overflow menu has reload variant, refresh PR status, add/remove from the queue, and open in browser.
• Status bar: runner phase and current QID. Click it for server actions (start/stop the container, reload, show logs, restart). Runner failures and manual-reload prompts appear as notifications with the relevant actions. Runner output goes to the PrairieLearn Review output channel.

Keyboard shortcuts

While focus is in one of the PrairieLearn Review views (and not in an input):

Cmd+Alt+P / Ctrl+Alt+P searches questions from anywhere while the console is running. You can rebind any of these with Preferences: Open Keyboard Shortcuts.

Settings

In attach mode, the PrairieLearn process must have the same course loaded as the workspace. If it is serving a different course, use managed mode or restart PrairieLearn with this course.

The queue state and runtime descriptor live under the platform user-state directory, keyed by the course's real path. Marking a question done saves a fingerprint of its question directory in that state. This baseline survives queue clearing and new reviews, so later edits can be selected with Changed since review. The add/new-review dialogs also provide Changed on branch, Unresolved comments, and Any review comments groups.

GitHub integration is read-only and optional. Local branch indicators use VS Code's Git support. PR identity comes from the official GitHub Pull Requests extension when available; after you choose Enable GitHub Pull Request Status, this extension uses the VS Code GitHub authentication provider to read changed-file and review-thread status from GitHub.com. Resolving or replying to comments remains in the official extension.

MCP configuration

Configure an MCP host to run:

node /absolute/path/to/dist/server/cli.js mcp \
  --course-root /path/to/course \
  --managed

If the extension (or a headless serve) is already running for that course, the MCP command attaches to it, and agent actions such as question_open move the preview and the views in VS Code. If nothing is running, it starts a temporary headless gateway and cleans it up when stdio closes. Standard output is reserved for MCP; diagnostics and child logs go to standard error.

Available tools cover server lifecycle, question discovery/search, exact-QID navigation, queue operations, tag catalog/read/update, course reload, and variant creation/reload. tags_update should first be called with dryRun: true; an apply call requires the returned baseHash as expectedHash.

Headless gateway

Without VS Code, serve runs only the API and PrairieLearn proxy:

node dist/server/cli.js serve --course-root /path/to/course --managed
node dist/server/cli.js serve --course-root /path/to/course --attach http://127.0.0.1:3000

Only one console (extension or serve) can run per course at a time.

Safety model

• The gateway binds to 127.0.0.1 and rejects unexpected Host headers (127.0.0.1 and localhost on its own port).
• Attach targets must resolve exclusively to loopback addresses.
• The review API requires a random bearer token. It is stored in a mode-0600 runtime descriptor for MCP. The extension calls the review service in-process and does not use the API.
• All question writes resolve through the discovered QID map and are restricted to known questions/<qid>/info.json files.
• Tag application uses a SHA-256 precondition and atomic replacement. Unknown catalog tags are preserved.
• Managed shutdown stops only the Docker container recorded by this console. Attach-mode processes are never signaled.
• To identify the rendered variant, the gateway reads the data-variant-id of question preview pages as they pass through. It requests those pages uncompressed.
• With plreview.relaxFrameHeaders, the gateway removes PrairieLearn's X-Frame-Options and frame-ancestors from proxied HTML so VS Code can frame it. Only use this with trusted local course content.

Development and testing

pnpm check         # type-check server, extension, and tests; knip
pnpm test          # vitest unit and loopback integration tests
pnpm test:vscode   # extension integration tests in a downloaded VS Code
pnpm build         # dist/server (CLI) and dist/extension.cjs

The loopback integration tests exercise API authorization, revision conflicts, cookies, redirects, form bodies, WebSocket upgrades, frame-header stripping, and variant observation. The VS Code suite starts a fake loopback PrairieLearn server (tests/fake-prairielearn.mjs) and runs the extension in attach mode against a temporary copy of the fixture course. It covers the views, the queue flow, preview variant observation, and tag drafts. It does not require Docker. On Linux it needs a display, so CI uses xvfb-run.