Bloblang Test Runner
A test runner for bento Bloblang/YAML test definitions, with two surfaces
sharing one execution story:
- a Go CLI (
blobl-test-runner) for automation/CI: list/run --json,
- a VS Code extension ("Bloblang Test Runner") with a Testing-view
explorer, Run and Debug profiles.
Runs execute the blobl_dbg engine as a Go library (headless); debug sessions
spawn the blobl-dbg binary serving DAP on stdio. Architecture and the locked
CLI JSON contract are in docs/adr/0001-runner-architecture.md;
terminology in GLOSSARY.md.
Prerequisites
Go and Node.js/npm on PATH — the repo ships a devbox.json
(go@latest, nodejs@latest, tea), so devbox shell provides them.
Sibling checkouts next to this repo (the runner's go.mod uses
filesystem replace directives that resolve as ../<name>):
bento — the hsctech fork on branch blobl-debug-hook
(carries the ParseWithDebug/DebugHook instrumentation),
blobl_dbg — codeberg.org/hsctech/blobl_dbg (package engine),
blobl_lsp — on branch issue-48-executable-lines
(the buildable variant: 2-arg ResolveImports + ExecutableLines),
blobl_sitter — codeberg.org/hsctech/blobl_sitter.
blobl-dbg binary (only needed for debugging, not for runs) — build it
from the blobl_dbg checkout and put it on PATH:
cd ../blobl_dbg && go build -o "$(go env GOPATH)/bin/blobl-dbg" ./cmd/blobl-dbg
Build & install the CLI
go build -o "$(go env GOPATH)/bin/blobl-test-runner" ./cmd/blobl-test-runner
# or: go install ./cmd/blobl-test-runner
blobl-test-runner list testdata --json # smoke check
CLI usage
blobl-test-runner list <file-or-dir> [--json]
blobl-test-runner run <file-or-dir> [--case <name>] [--json]
Discovery: a .yaml/.yml file is a test definition iff it has a
non-empty top-level tests: list. Companion files win over inline tests:
(foo_bento_test.yaml / foo_test.yaml pair with foo.yaml), and
_bento_test beats _test; a suffix file with an empty tests: is skipped.
The launch target is the companion config when present, else the file
itself. Case id = <definition-file>#<name>; unnamed cases are test <i>
(1-based), duplicates get #2, #3, … --case selects the first case
with that name. JSON output paths are always absolute; directory runs are
lexical; run exits 0 when all selected cases pass, 1 otherwise (2 = usage
error). Without --json, output is human-readable.
Examples over the bundled fixtures (which include intentional failures):
blobl-test-runner list testdata # human-readable
blobl-test-runner list testdata --json # {definitions: [{file, launch, error?, cases: [{name, target?}]}]}
blobl-test-runner run testdata --json; echo $? # 1 — dup#2, "real jq fails", "shouts failing" fail with detail; "missing mapping" is a case-level error
blobl-test-runner run testdata/pipeline_bento_test.yaml --case "mocked jq passes" --json; echo $? # 0
blobl-test-runner run testdata/dup_test.yaml --case dup --json; echo $? # 0 — first match only ("dup#2" not run)
run --json shape:
{runs: [{definition, launch, failed, error?, cases: [{name, passed, durationMs, error?, batches: [{batch, countErr?, messages: [{err?, conditions: [{kind, pass, detail?}]}]}]}]}]}.
The run-level error? is reserved for definition-level errors (malformed
definition, unknown --case, launch failure); a case that cannot run at all
(e.g. a missing target_mapping) reports passed: false with its own
error? on the case entry, and the other cases still run.
Verification
go test ./...
npm install && npm run typecheck && npm run build
scripts/e2e.sh # builds the CLI from source and asserts list/run behavior over testdata/,
# then runs scripts/dap-smoke.mts (DAP debug launch over blobl-dbg)
npm run smoke # exercises the extension's CLI client (src/cli.ts) against a built binary
scripts/e2e.sh is the end-to-end check: from a clean checkout it builds the
CLI and asserts the list JSON shape + absolute paths, that run testdata
exits 1 with the failing cases marked and detail present, that a single
passing --case exits 0, that --case is first-match by name, and that a
case-level error (missing target_mapping) lands on the case entry only. It
then runs scripts/dap-smoke.mts, which locates or builds blobl-dbg (built
from the sibling blobl_dbg checkout via a throwaway go.work in /tmp that
pins blobl_lsp to the issue-48 worktree — blobl_dbg itself is untouched;
fails loudly if no binary can be produced) and drives blobl-dbg dap over
stdio with a minimal DAP client: a bloblang-test launch
({mapping, testFile, testName}) must show one entry stop, continue must
work, a failing case must emit per-condition FAIL output events, and the
session must end with exited exitCode 1 (0 for a passing case) followed by
terminated. Run it standalone with node scripts/dap-smoke.mts [testdata]
(BLOBL_DBG=/path/to/blobl-dbg to use an existing binary).
CI & releases
Forgejo Actions workflows live in .forgejo/workflows/ (same pattern as the
sibling blobl_dbg / blobl_lsp_vscode repos):
ci.yml — on pushes/PRs to main: a Go job clones the repo plus its
replace-deps as siblings (blobl_dbg, bento @ blobl-debug,
blobl_lsp @ issue-48-executable-lines, blobl_sitter) and runs
gofmt / go vet / go test / go build / scripts/e2e.sh (CLI
assertions + DAP smoke against a fresh blobl-dbg build); a Node job runs
npm ci, npm run typecheck, npm test, npm run build.
release.yml — on version tags (v*): a copy of blobl_lsp_vscode's
publish.yml. Verifies the tag matches package.json's version, runs
npm ci / npm test, packages the extension VSIX (npm run package),
attaches it to the Codeberg release for the tag
(actions/forgejo-release@v2), then publishes to the VS Code Marketplace
(npx vsce publish, needs the VSCE_PAT secret) and Open VSX
(npx ovsx publish, needs the OVSX_PAT secret).
To cut a release: bump version in package.json, commit, then push a tag
matching it (git tag v0.1.0 && git push origin v0.1.0).
VS Code usage
Build the extension once:
npm install && npm run build # esbuild -> dist/extension.js
Settings (Settings UI or settings.json):
| Setting |
Default |
Meaning |
blobltest.runner.path |
blobl-test-runner |
CLI binary used for discovery and runs (looked up on PATH when relative). |
blobltest.debugger.path |
"" |
blobl-dbg binary for debug sessions. Empty resolves as: managed download in global storage (see below) first, then blobl-dbg on PATH. |
Debugger binary resolution (ADR-0001 decision 3), in precedence order:
blobltest.debugger.path when set to an existing file;
- the managed download in the extension's global storage —
<globalStorage>/debugger/<release-tag>/blobl_dbg_<os>_<arch>[.exe]
(e.g. ~/.config/Code/User/globalStorage/hsctech-dev.blobl-test-runner/debugger/v0.0.4/blobl_dbg_linux_amd64);
- a bare
blobl-dbg on PATH.
Run the command Bloblang Tests: Update Debugger to (re)download the pinned
blobl_dbg release into that managed location (checksum-verified against the
release's SHA256SUMS). If the binary is found nowhere — on the explorer
debug path or a direct launch.json "type": "bloblang-test" launch — you
get the error notification "blobl-dbg not found: install it or set
blobltest.debugger.path" with an Update Debugger action, and the debug
session aborts.
Using it:
- Open a folder that contains test definitions (e.g. this repo, which
has
testdata/).
- Open the Testing view: the Bloblang Test Runner controller lists
folder → definition file → test case. Cases with a
target_mapping are
labeled name -> map.blobl. Press the refresh icon (or run
Bloblang Tests: Refresh) to re-discover.
- Run tests: the "Run tests" profile — play button on the controller,
folder, file, or case, or the inline run icon on an item. Results appear
in the Test Results panel; failing cases show the failing condition's
detail (e.g. the JSON mismatch diff).
- Debug a failing case: the inline debug icon (or the "Debug tests"
profile) on a case item. This synthesizes a
bloblang-test launch
(mapping = launch target, testFile = definition file, testName =
case name, all absolute) and spawns blobl-dbg dap. A case = one case in
one session; a file = all its cases in one session (testName: ""); a
folder = one session per definition file, lexical order. Debug results
are reported only — they never mark TestItems. If no debugger binary is
found you get "blobl-dbg not found: install it or set
blobltest.debugger.path" (with an Update Debugger action) and the
session aborts — this covers direct launch.json "type": "bloblang-test" launches too, since the debug adapter factory resolves
the binary with the same setting > managed > PATH precedence.
A manual test debug launch looks like:
{
"type": "bloblang-test",
"request": "launch",
"name": "Debug pipeline tests",
"mapping": "/abs/path/to/pipeline.yaml", // launch target (.blobl or stream config)
"testFile": "/abs/path/to/pipeline_bento_test.yaml",
"testName": "", // "" = all cases; a name = first match
// "input": "/abs/path/input.jsonl", // optional fallback input for cases without batches
// "mockSteps": [2] // optional 1-based pipeline steps to mock
}
Extension Development Host smoke steps
npm install && npm run build
code --extensionDevelopmentPath="$PWD" "$PWD" # opens an Extension Development Host window
In the Development Host window:
- Open the Testing view — the Bloblang Test Runner controller is
present; run
Bloblang Tests: Refresh and confirm the testdata/
definitions appear (6 files, including malformed_test.yaml with an
error badge, and no pipeline_test.yaml/skipped_test.yaml).
- Run all on the workspace folder: passing cases go green, the known
failures (
dup#2, real jq fails, shouts failing, missing mapping)
go red with detail in the Test Results panel.
- Debug a failing case (e.g.
real jq fails): a bloblang-test debug
session starts via blobl-dbg dap (needs blobl-dbg on PATH or
blobltest.debugger.path set), output conditions are reported in the
Debug Console, and the session exits.
- If the CLI binary is not on the EDH's PATH, set
blobltest.runner.path
to the absolute path of the built blobl-test-runner and refresh.
Layout
├── cmd/blobl-test-runner/ # Go CLI: list/run --json (engine-as-library)
├── src/ # extension: extension.ts (explorer + profiles), cli.ts (CLI client)
├── scripts/ # e2e.sh (CLI + DAP end-to-end), smoke-cli.mts (CLI-client smoke), dap-smoke.mts (DAP smoke)
├── testdata/ # fixtures incl. intentional failures
├── docs/adr/0001-runner-architecture.md
└── GLOSSARY.md