AutoJudge Code Runner

Run, test, and evaluate programming solutions directly within Visual Studio Code.

AutoJudge brings the power of competitive programming evaluation and solution testing straight to your IDE. With zero overhead, you can run files against the AutoJudge API or connect to a self-hosted server instance. Test solutions instantly using local input/output files and watch the execution results stream in real time.

[!TIP] Contests on VS Code - Use AutoJudge Contest extension to participate in live programming contests directly from your editor, leveraging the same AutoJudge API for seamless integration.

Execution Logs

Features

⚡ Zero-Config Code Running: Run your active code file against the remote judge instantly without managing compiler or interpreter paths locally.
🧪 Local Testcase Suite: Automatically find and run your solution against multiple test inputs (.in files) and verify correctness.
🔍 Diff-Based Test Validation: Assert correctness by comparing program outputs to expected .out files with detailed diagnostics on mismatch.
🔄 Auto-Discovery: Smart automatic detection of test cases in your source directory or a custom path.
🌐 Self-Hosting Support: Run your code against private AutoJudge servers for local development and testing (Check @werlang/autojudge)
📊 Integrated UX: Clear output channel reports with execution details, execution times, diff status, and visual status badges.

Comparison of Execution Modes

Feature / Mode	⚡ Code Runner Mode	🧪 Test Mode
Command	`AutoJudge: Run Active File (Code Runner)`	`AutoJudge: Test Active File (Test Mode)`
Target Audience	Quick debugging / experimental executions	Rigorous verification of edge cases and logic
Output Checking	Ignored; displays raw output	Strictly validates matches with expected `.out` files
Prerequisites	Needs code file and optional inputs	Requires matching `.out` for every `.in` file
Failure Behavior	Fails on runtime or server errors	Fails early if any `.out` pair is missing

Supported Languages & Extensions

AutoJudge automatically matches active editors based on their file extension:

Language	Extension
C	`.c`
C++	`.cpp`
Java	`.java`
JavaScript	`.js`
PHP	`.php`
Python	`.py`
TypeScript	`.ts`

Install

Get the extension using one of the following methods:

🛒 Visual Studio Marketplace

Open VS Code and navigate to the Extensions View (Ctrl+Shift+X or Cmd+Shift+X).
Search for AutoJudge.
Click Install.

⌨️ Command Palette (Quick Open)

Launch Quick Open (Ctrl+P or Cmd+P), paste the following command, and press Enter:

ext install autojudge.autojudge-extension

💻 Command Line (CLI)

You can install via terminal:

code --install-extension autojudge.autojudge-extension

Quick Start

Get up and running in three simple steps:

1. Structure Your Test Files

Create your code file alongside .in files (inputs) and matching .out files (outputs) in the same directory:

my-workspace/
├── solution.py
├── test1.in        # Input case 1
├── test1.out       # Expected output 1
├── test2.in        # Input case 2
└── test2.out       # Expected output 2

Note: If no .in files are present, AutoJudge defaults to sending a single empty-string input.

2. Run the Solution

Open your code file (e.g. solution.py) in the editor:

Click the Play ⚡ (Code Runner) or Test 🧪 (Test Mode) icon in the top-right editor title bar.
Or use shortcuts: Ctrl+Alt+R (Run) / Ctrl+Alt+T (Test).

3. Review Results

The extension redirects focus to the AutoJudge output channel, printing test progress and final verdicts:

Execution Logs

Commands

Find these commands via the Command Palette (Ctrl+Shift+P / Cmd+Shift+P / F1):

Command Title	Command ID	Description
`AutoJudge: Run Active File (Code Runner)`	`autojudge.runActiveFile`	Run code against inputs, ignoring `.out` files.
`AutoJudge: Test Active File (Test Mode)`	`autojudge.testActiveFile`	Test code against inputs, strictly verifying against `.out` files.

Both commands are easily accessible from the editor title button bar: Editor Icons

Keyboard Shortcuts

Run solutions instantly using default keybindings:

Action	OS (Windows / Linux)	macOS
⚡ Run Active File	`Ctrl` + `Alt` + `R`	`Ctrl` + `Alt` + `R`
🧪 Test Active File	`Ctrl` + `Alt` + `T`	`Ctrl` + `Alt` + `T`

Editor Action Bar

Configuration

Fine-tune the extension's behavior in your settings.json:

Setting Key	Type	Default Value	Description
`autojudge.baseUrl`	`string`	`"https://api.autojudge.io"`	The full endpoint URL of the AutoJudge server API.
`autojudge.testcasePath`	`string`	`""` (Use source dir)	Directory containing testcases. Supports VS Code variables like `${workspaceFolder}` and `${fileDirname}`.

Configuration Example

{
  "autojudge.baseUrl": "https://api.autojudge.io",
  "autojudge.testcasePath": "${fileDirname}/testcases"
}

Troubleshooting

Mismatched Test Files (Test Mode)

Issue: Test Mode displays an error without running the code. Fix: Test Mode requires strict pairing. Verify that every .in file has a corresponding same-name .out file (e.g. case01.in and case01.out) in the testcase directory.

Custom Host Connection Refused

Issue: Connection times out or errors during evaluation. Fix: Verify your autojudge.baseUrl setting. Test that the host is accessible from your network and that the API endpoint conforms to the AutoJudge schema.

Empty Inputs Running by Default

Issue: The execution ran with "empty input". Fix: By design, if no .in files are detected in the active folder or testcasePath, the extension sends a single empty string payload as input.

Release Notes

For detailed updates and changes, please refer to our CHANGELOG.md.

Development

If you want to contribute to the extension's development, set up the containerized environment:

Build and run the service container:
```
docker compose up -d --build
```
Install Node dependencies inside the running container:
```
docker compose exec extension npm install
```
Run the Vitest test suites:
```
docker compose exec extension npm test
```