Socratic Code Hinter
A VS Code extension that helps students debug their own Python code through Socratic hints, quizzes, fault localization, test generation, and memory-graph debugging.
This repository is a monorepo with four parts:
| Component |
What it does |
src/ + dist/ |
The VS Code extension (TypeScript) |
src/webview/ |
The sidebar UI (Svelte) |
api/ |
Express API for GitHub auth, sessions, and vector search |
relay-api/ |
FastAPI relay for AI inference and quota enforcement |
1. Run it locally
This is the fastest path for someone who just wants to start the project on their machine.
1.1 Prerequisites
- VS Code with the built-in “Run Extension” debugging capability
- Node.js 20.20.1 and npm
- Python 3.10+
- uv — only needed for the local relay backend
1.2 One-time setup
git clone https://github.com/mattChrisP/socracodehinter.git
cd socracodehinter
# Install extension dependencies
npm ci
# Install webview dependencies
npm ci --prefix src/webview
# Install relay dependencies (optional, only if running relay-api locally)
cd relay-api
uv sync --extra dev
cd ..
Copy the example environment files and fill in the values.
# API (sessions + GitHub login)
test -f api/.env || cp api/example.env api/.env
# Relay (AI inference)
test -f relay-api/.env || cp relay-api/.env.example relay-api/.env
api/.env requires at least:
GITHUB_CLIENT_ID=...
GITHUB_CLIENT_SECRET=...
ACCESS_TOKEN_SECRET=...
SUPABASE_URL=...
SUPABASE_SERVICE_ROLE_KEY=...
NODE_ENV=development
DEV_API_URL=http://localhost:5001
relay-api/.env requires at least:
OPENROUTER_API_KEY=...
JINA_API_KEY=...
ACCESS_TOKEN_SECRET=... # at least 32 bytes
SUPABASE_DB_URL=...
1.4 Start everything
Open the repo in VS Code, then run the services in separate terminals.
Terminal 1 — local API:
npm run dev:api
Expected output:
Server listening on port 5001
Terminal 2 — local relay:
npm run dev:relay
This starts the FastAPI relay on http://localhost:8080.
VS Code — launch the extension:
- Press
F5.
- Choose the Local launch configuration.
- A new “Extension Development Host” window opens.
The checked-in .vscode/launch.json Local configuration already points the extension at:
| Service |
URL |
| Session API |
http://localhost:5001 |
| Relay root |
http://localhost:8080 |
| Embeddings |
http://localhost:8080/v1/embeddings |
If you only want to use the hosted production backend, choose the Prod launch configuration and skip both terminals above.
Tip: npm run dev starts the extension watchers and API together. It does not start the relay — use npm run dev:relay for that.
1.5 Sign in
In the Extension Development Host:
- Open the Socratic Code Hinter sidebar.
- Click Login with GitHub.
- Complete the browser flow and return to VS Code.
On first launch the extension also creates an isolated Python environment for pytest, pylint, black, ipykernel, and the bundled memory graph module. This can take a minute the first time.
2. Install for normal use (VSIX)
Users who only want to use the extension do not need this repository.
Option A — VS Code UI
- Open VS Code.
- Go to Extensions.
- Click
... → Install from VSIX....
- Select
codehinter.vsix.
Option B — command line
code --install-extension codehinter.vsix
Remove a previous install with:
code --uninstall-extension socracodelabs.socratic-code-hinter
For isolated testing:
code --user-data-dir ./test-profile --extensions-dir ./test-extensions
3. Project layout
| Path |
Purpose |
src/extension.ts |
Extension activation, commands, debug/memory-graph wiring |
src/controller.ts |
Editor, Python, pytest/pylint, test generation, fault localization |
src/providers/ |
Webview provider, auth, token/session managers, memory graph |
src/graph/ |
LangGraph state machines for pedagogical workflows |
src/utils/ |
Prompt loading, model routing, session/API clients, subprocess helpers |
src/webview/src/ |
Svelte sidebar UI and message bridge |
api/ |
Express server and routes |
relay-api/ |
FastAPI relay, quota layer, upstream clients |
python_vendor/ |
Bundled Python packages such as memory_graph and the SBFL pytest plugin |
promptfoo/ |
Prompt evaluation configs and fixtures |
4. Common commands
| Task |
Command |
| Local API |
npm run dev:api |
| Local relay |
npm run dev:relay |
| Extension + webview + API watchers |
npm run dev |
| Unit tests |
npm run unit_test |
| Integration tests |
npm run test:integration |
| E2E tests |
npm run test:e2e |
| Relay tests |
npm run test:relay |
| Typecheck |
npm run typecheck |
| Lint |
npm run lint |
| Build webview + extension |
npm run build:ext |
| Package VSIX |
npm run package:vsix |
| Install the packaged VSIX |
npm run install:vsix |
5. Configuration
Extension launch configs
Environment for the extension host is configured in .vscode/launch.json, not a root .env file.
- Prod uses the hosted API and relay.
- Local uses
http://localhost:5001 and http://localhost:8080.
Useful extension-host variables:
| Variable |
Purpose |
NODE_ENV |
development switches API URL resolution to DEV_API_URL |
DEV_API_URL |
Local Express API URL |
FASTAPI_LOCAL_RELAY |
Relay path, e.g. http://localhost:8080/v1 |
FASTAPI_LOCAL_RELAY_ROOT |
Relay root, e.g. http://localhost:8080 |
FASTAPI_LOCAL_EMB |
Embedding endpoint override |
SCH_DEBUG_LOGS |
true/1/yes/on enables debug logging |
Debug logs
Set SCH_DEBUG_LOGS=true in the active launch config, then open the VS Code Debug Console. This logs graph flow, model routing, subprocess timing, and memory-graph setup.
Error reporting
The extension catches webview and extension-host errors, asks before sending (socratic-code-hinter.errorReporting.mode defaults to prompt), and posts sanitized reports to the API's /error-report route.
- Issue creation is dry-run by default: the API stores reports in Supabase but creates no GitHub issues until
ERROR_REPORT_DRY_RUN=false and GITHUB_ISSUE_TOKEN are set. If the error_reports table is unavailable, the route falls back to creating the issue without dedupe and returns dedupeAvailable: false.
- Reports are deduplicated by error fingerprint, so repeated occurrences update one issue instead of opening duplicates.
- Student code is attached only when the student picks “Report with code”, or when the school configures
includeCode: always with mode: auto.
- See
api/README.md for the error_reports table SQL.
- With
SCH_DEBUG_LOGS=true, the Command Palette command Debug: Trigger Error Reporting Scenario runs each report path without editing source.
6. Building the VSIX
npm run package:vsix
This creates codehinter.vsix at the repo root.
Packaging security checks:
unzip -l codehinter.vsix | grep -iE 'relay|\.env' # must print nothing
The package must never contain api/, relay-api/, or secret files.
7. What the extension can do
The UI is the Socratic Code Hinter sidebar.
| User action |
What happens |
| Run End-to-End Test |
Runs syntax/pylint/pytest and guides the student through the fix |
| Run Python File |
Executes the active Python file or selected notebook cells |
| Generate Test Cases |
Creates or extends a pytest file |
| Locate Lines with Possible Error |
Runs SBFL and highlights suspicious lines |
| Provide Hint and Quiz |
Generates a Socratic multiple-choice hint |
| Insert Print Variables |
Instruments the code with diagnostic print() calls |
| Visualize Memory Graph |
Starts a Python debug session and renders memory_graph |
| Descriptive Pseudocode Solution |
RAG + LLM generates pseudocode, not copyable code |
| Provide Code Solution |
Shows a gated side-by-side solution preview |
The LLM never executes tools. The controller runs pytest/pylint/SBFL/RAG itself and passes their reports to the model as evidence.
8. Backend architecture in one page
API (api/)
- Express + Supabase
- GitHub OAuth issues JWTs
- Session and message persistence
- Vector search endpoints
- Effort-gated solution access: solution lookups require an owned session with at least one persisted human attempt
Relay (relay-api/)
- FastAPI proxy to OpenRouter and Jina
Authorization: Bearer <jwt> on every route- Per-user quota: estimate → atomic consume → refund on failure
- Request-size caps and defensive upstream response validation
/healthz liveness and /readyz database readiness
RAG pipeline
- Canonicalize student code locally with
src/utils/canonicalize.py
- Embed with Jina
- Search Supabase
pgvector for similar solutions
- Use the result as context for the model — never return raw answer dumps
Model routing
Route keys and default tiers live in src/utils/modelRouting.ts. The relay resolves tiers to concrete OpenRouter model IDs in relay-api/relay_api/routers/inference.py and disables reasoning (reasoning.enabled=false) to keep replies fast and non-thinking.
9. Testing
| Layer |
Command |
Notes |
| Unit |
npm run unit_test |
Vitest, mocks vscode |
| Integration |
npm run test:integration |
Real VS Code test host |
| E2E |
npm run test:e2e |
Uses .vscodeignore.e2e and a local mock backend |
| API |
npm --prefix api test |
Vitest + supertest |
| Relay |
npm run test:relay |
Full mocked pytest suite, including stress/benchmarks |
10. Prompt evaluation
Before changing prompts:
npm run promptfoo:eval:chat
npm run promptfoo:eval:error
npm run promptfoo:view
Prompt templates ship under promptfoo/prompts/ and are loaded by src/utils/promptLoader.ts.
11. Troubleshooting
Usually an authentication issue.
- Click Login with GitHub.
- If already logged in, run
Developer: Reload Window.
Port already in use
lsof -i :5001
lsof -i :8080
Then stop the conflicting process or change the port.
Memory graph does not appear
- Check the debug session produced
memory_graph.svg.
- Confirm
dist/viz/graphviz.js, dist/viz/render_graph_svg.mjs, and python_vendor/memory_graph/ exist.
- Rebuild with
npm run build:ext.
- Native Graphviz is optional; the bundled JavaScript renderer is used automatically.
Python environment is broken
- Open the Command Palette (
Cmd/Ctrl + Shift + P).
- Run Socratic Code Hinter: Reset Internal Python Environment.
- Reload VS Code.
Login or token errors
- Clear the stored token with Socratic Code Hinter: Clear Token.
- Sign in again with GitHub.
- Expired sessions now reset the sidebar to the GitHub login screen automatically.
- For local GitHub login, make sure the GitHub OAuth app callback URL matches the local API, for example
http://localhost:5001/auth/github/callback.
12. Deploying / operating
Production deployment details are split by component:
api/Procfile → Heroku web processrelay-api/Procfile → Heroku web processscripts/heroku-scale-up.sh → wake/scaling helper
See api/README.md and relay-api/README.md for component-specific deployment notes.
13. Developer conventions
See AGENTS.md for product boundaries, architecture rules, prompt requirements, and the full feature-to-file map.
