AutoJudge Code Runner
Run your solution against your own test cases from VS Code, on the AutoJudge judge. Write .in and .out files next to your code, press one button, and see each case's output or verdict in the editor. You don't need compilers or interpreters on your machine.
Features
- Run or test in one click. Run shows what your program prints for every
.in file. Test compares the output with the matching .out file and gives a verdict for each case.
- Results you can scan. Each case starts with ✓ or ✗, its file name, its verdict, and its run time, in aligned columns. Passes show in green and failures in red in the default themes. Wrong answers show the expected and received output one above the other, with the first line that differs.
- Runtime errors and time limits are named. A crash shows the error message the judge returned, indented under its case like your program's output, and a summary line counts the cases that failed.
- Uses your test files. Every
.in file next to your code (or in a folder you choose) is a test case. Cases run in natural order, so case2.in runs before case10.in.
- Catches mistakes before sending. Test mode stops if a
.out file is missing and names it. File names the judge would reject are caught first, with a suggested new name. Only one run happens at a time.
- Works with your own server. Point the extension at a self-hosted AutoJudge server.
[!TIP]
Competing in an AutoJudge contest? AutoJudge Contest lets you read problems, export their public test cases as .in / .out files, and submit from VS Code. Use this extension to test against those files before you submit.
Supported languages
| Language |
Extension |
| C |
.c |
| C++ |
.cpp |
| Java |
.java |
| JavaScript |
.js |
| PHP |
.php |
| Python |
.py |
Install
Search for AutoJudge Code Runner in the Extensions view (Ctrl+Shift+X / Cmd+Shift+X) and click Install, or run:
code --install-extension autojudge.autojudge-extension
Quick Start
Put your solution and its test cases in one folder. Each .in file is one input; the .out file with the same name is its expected output.
two-sum/
├── solution.py
├── case1.in
├── case1.out
├── case2.in
└── case2.out
Open solution.py and click Run (play icon) or Test (beaker icon) in the editor title bar.
Read the result in the AutoJudge output panel. What your program printed is indented under each case, and the status bar shows a short summary for a few seconds. Click the folder path (Cmd+click / Ctrl+click) to show it in the Explorer.
Unsaved changes are included in the run. If the folder has no .in files, Run runs your program once with empty input.
Commands
| Command |
Command ID |
What it does |
| AutoJudge: Run Active File |
autojudge.runActiveFile |
Runs the file with each .in file as input and shows what it prints. .out files are not used. |
| AutoJudge: Test Active File |
autojudge.testActiveFile |
Runs the file with each .in file and compares the output with the matching .out file. Every .in file needs a .out file. |
Both commands appear in the editor title bar and the Command Palette for .c, .cpp, .java, .js, .php, and .py files saved on disk. One run happens at a time; cancel a run from its progress notification.
Keyboard Shortcuts
| Command |
Windows / Linux |
macOS |
| AutoJudge: Run Active File |
Ctrl+Alt+R |
Ctrl+Option+R |
| AutoJudge: Test Active File |
Ctrl+Alt+T |
Ctrl+Option+T |
Configuration
| Setting |
Default |
Description |
autojudge.baseUrl |
https://api.autojudge.io |
AutoJudge API address. Change it only if you use your own server. Base paths such as https://example.com/api are supported. |
autojudge.testcasePath |
"" |
Folder with your test cases. Leave empty to use the folder of the file you run. Supports ${workspaceFolder}, ${fileDirname}, and other file variables. Relative paths start from the folder of the file you run. |
{
"autojudge.testcasePath": "${fileDirname}/tests"
}
Only .in and .out files directly inside the folder are used; subfolders are ignored. A testcasePath that does not exist or is not a folder stops the run with an error and an Open Settings button.
Troubleshooting
"Test mode needs a .out file for every .in file. Missing …" — Add the missing .out file next to its .in file, or click Run Instead to run without comparing outputs.
"Test mode compares each .in file with its .out file. No .in files found in …" — Add .in / .out pairs to that folder or set autojudge.testcasePath, or click Run Instead.
"Rename … to run it on AutoJudge" — The judge only accepts file names made of letters, digits, _ and -, such as two_sum.py. The extension also asks for a lowercase extension, because older judges hang on names like Main.PY.
"Could not reach the AutoJudge server …" or "did not answer within 20 seconds" — Check your connection and autojudge.baseUrl (click Open Settings). It must be the API address, which is not always the same as the website address.
"The outputs differ only in whitespace" — The judge ignores leading and trailing whitespace but compares spaces and line breaks inside the output. Check for extra spaces between values or extra blank lines. Windows line endings in .in and .out files are converted for you.
"Run failed" followed by "The judge returned results for 2 of 3 cases." — The judge stopped partway through, usually because the whole run took too long. Try again, or run fewer cases at once.
Release Notes
See the CHANGELOG.
Contributing
Issues and pull requests are welcome on GitHub. Development runs in Docker Compose:
docker compose up -d --build
docker compose exec extension npm install
docker compose exec extension npm test
Press F5 in VS Code to launch the Extension Development Host.
License
MIT