Skip to content
| Marketplace
Sign in
Visual Studio Code>Testing>Bloblang Test RunnerNew to Visual Studio Code? Get it now.
Bloblang Test Runner

Bloblang Test Runner

hsctech

| (0) | Free
Test explorer for bento bloblang/YAML test definitions, with run and debug profiles (blobl-dbg DAP).
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

Bloblang Test Runner

A test runner for bento Bloblang/YAML test definitions, with two surfaces sharing one execution story:

  • a Go CLI (blobl-test-runner) for automation/CI: list/run --json,
  • a VS Code extension ("Bloblang Test Runner") with a Testing-view explorer, Run and Debug profiles.

Runs execute the blobl_dbg engine as a Go library (headless); debug sessions spawn the blobl-dbg binary serving DAP on stdio. Architecture and the locked CLI JSON contract are in docs/adr/0001-runner-architecture.md; terminology in GLOSSARY.md.

Prerequisites

  • Go and Node.js/npm on PATH — the repo ships a devbox.json (go@latest, nodejs@latest, tea), so devbox shell provides them.

  • Sibling checkouts next to this repo (the runner's go.mod uses filesystem replace directives that resolve as ../<name>):

    • bento — the hsctech fork on branch blobl-debug-hook (carries the ParseWithDebug/DebugHook instrumentation),
    • blobl_dbg — codeberg.org/hsctech/blobl_dbg (package engine),
    • blobl_lsp — on branch issue-48-executable-lines (the buildable variant: 2-arg ResolveImports + ExecutableLines),
    • blobl_sitter — codeberg.org/hsctech/blobl_sitter.
  • blobl-dbg binary (only needed for debugging, not for runs) — build it from the blobl_dbg checkout and put it on PATH:

    cd ../blobl_dbg && go build -o "$(go env GOPATH)/bin/blobl-dbg" ./cmd/blobl-dbg
    

Build & install the CLI

go build -o "$(go env GOPATH)/bin/blobl-test-runner" ./cmd/blobl-test-runner
# or: go install ./cmd/blobl-test-runner
blobl-test-runner list testdata --json   # smoke check

CLI usage

blobl-test-runner list <file-or-dir> [--json]
blobl-test-runner run  <file-or-dir> [--case <name>] [--json]

Discovery: a .yaml/.yml file is a test definition iff it has a non-empty top-level tests: list. Companion files win over inline tests: (foo_bento_test.yaml / foo_test.yaml pair with foo.yaml), and _bento_test beats _test; a suffix file with an empty tests: is skipped. The launch target is the companion config when present, else the file itself. Case id = <definition-file>#<name>; unnamed cases are test <i> (1-based), duplicates get #2, #3, … --case selects the first case with that name. JSON output paths are always absolute; directory runs are lexical; run exits 0 when all selected cases pass, 1 otherwise (2 = usage error). Without --json, output is human-readable.

Examples over the bundled fixtures (which include intentional failures):

blobl-test-runner list testdata                          # human-readable
blobl-test-runner list testdata --json                   # {definitions: [{file, launch, error?, cases: [{name, target?}]}]}
blobl-test-runner run testdata --json; echo $?           # 1 — dup#2, "real jq fails", "shouts failing" fail with detail; "missing mapping" is a case-level error
blobl-test-runner run testdata/pipeline_bento_test.yaml --case "mocked jq passes" --json; echo $?   # 0
blobl-test-runner run testdata/dup_test.yaml --case dup --json; echo $?                              # 0 — first match only ("dup#2" not run)

run --json shape: {runs: [{definition, launch, failed, error?, cases: [{name, passed, durationMs, error?, batches: [{batch, countErr?, messages: [{err?, conditions: [{kind, pass, detail?}]}]}]}]}]}. The run-level error? is reserved for definition-level errors (malformed definition, unknown --case, launch failure); a case that cannot run at all (e.g. a missing target_mapping) reports passed: false with its own error? on the case entry, and the other cases still run.

Verification

go test ./...
npm install && npm run typecheck && npm run build
scripts/e2e.sh        # builds the CLI from source and asserts list/run behavior over testdata/,
                      # then runs scripts/dap-smoke.mts (DAP debug launch over blobl-dbg)
npm run smoke         # exercises the extension's CLI client (src/cli.ts) against a built binary

scripts/e2e.sh is the end-to-end check: from a clean checkout it builds the CLI and asserts the list JSON shape + absolute paths, that run testdata exits 1 with the failing cases marked and detail present, that a single passing --case exits 0, that --case is first-match by name, and that a case-level error (missing target_mapping) lands on the case entry only. It then runs scripts/dap-smoke.mts, which locates or builds blobl-dbg (built from the sibling blobl_dbg checkout via a throwaway go.work in /tmp that pins blobl_lsp to the issue-48 worktree — blobl_dbg itself is untouched; fails loudly if no binary can be produced) and drives blobl-dbg dap over stdio with a minimal DAP client: a bloblang-test launch ({mapping, testFile, testName}) must show one entry stop, continue must work, a failing case must emit per-condition FAIL output events, and the session must end with exited exitCode 1 (0 for a passing case) followed by terminated. Run it standalone with node scripts/dap-smoke.mts [testdata] (BLOBL_DBG=/path/to/blobl-dbg to use an existing binary).

CI & releases

Forgejo Actions workflows live in .forgejo/workflows/ (same pattern as the sibling blobl_dbg / blobl_lsp_vscode repos):

  • ci.yml — on pushes/PRs to main: a Go job clones the repo plus its replace-deps as siblings (blobl_dbg, bento @ blobl-debug, blobl_lsp @ issue-48-executable-lines, blobl_sitter) and runs gofmt / go vet / go test / go build / scripts/e2e.sh (CLI assertions + DAP smoke against a fresh blobl-dbg build); a Node job runs npm ci, npm run typecheck, npm test, npm run build.
  • release.yml — on version tags (v*): a copy of blobl_lsp_vscode's publish.yml. Verifies the tag matches package.json's version, runs npm ci / npm test, packages the extension VSIX (npm run package), attaches it to the Codeberg release for the tag (actions/forgejo-release@v2), then publishes to the VS Code Marketplace (npx vsce publish, needs the VSCE_PAT secret) and Open VSX (npx ovsx publish, needs the OVSX_PAT secret).

To cut a release: bump version in package.json, commit, then push a tag matching it (git tag v0.1.0 && git push origin v0.1.0).

VS Code usage

Build the extension once:

npm install && npm run build     # esbuild -> dist/extension.js

Settings (Settings UI or settings.json):

Setting Default Meaning
blobltest.runner.path blobl-test-runner CLI binary used for discovery and runs (looked up on PATH when relative).
blobltest.debugger.path "" blobl-dbg binary for debug sessions. Empty resolves as: managed download in global storage (see below) first, then blobl-dbg on PATH.

Debugger binary resolution (ADR-0001 decision 3), in precedence order:

  1. blobltest.debugger.path when set to an existing file;
  2. the managed download in the extension's global storage — <globalStorage>/debugger/<release-tag>/blobl_dbg_<os>_<arch>[.exe] (e.g. ~/.config/Code/User/globalStorage/hsctech-dev.blobl-test-runner/debugger/v0.0.4/blobl_dbg_linux_amd64);
  3. a bare blobl-dbg on PATH.

Run the command Bloblang Tests: Update Debugger to (re)download the pinned blobl_dbg release into that managed location (checksum-verified against the release's SHA256SUMS). If the binary is found nowhere — on the explorer debug path or a direct launch.json "type": "bloblang-test" launch — you get the error notification "blobl-dbg not found: install it or set blobltest.debugger.path" with an Update Debugger action, and the debug session aborts.

Using it:

  1. Open a folder that contains test definitions (e.g. this repo, which has testdata/).
  2. Open the Testing view: the Bloblang Test Runner controller lists folder → definition file → test case. Cases with a target_mapping are labeled name -> map.blobl. Press the refresh icon (or run Bloblang Tests: Refresh) to re-discover.
  3. Run tests: the "Run tests" profile — play button on the controller, folder, file, or case, or the inline run icon on an item. Results appear in the Test Results panel; failing cases show the failing condition's detail (e.g. the JSON mismatch diff).
  4. Debug a failing case: the inline debug icon (or the "Debug tests" profile) on a case item. This synthesizes a bloblang-test launch (mapping = launch target, testFile = definition file, testName = case name, all absolute) and spawns blobl-dbg dap. A case = one case in one session; a file = all its cases in one session (testName: ""); a folder = one session per definition file, lexical order. Debug results are reported only — they never mark TestItems. If no debugger binary is found you get "blobl-dbg not found: install it or set blobltest.debugger.path" (with an Update Debugger action) and the session aborts — this covers direct launch.json "type": "bloblang-test" launches too, since the debug adapter factory resolves the binary with the same setting > managed > PATH precedence.

A manual test debug launch looks like:

{
  "type": "bloblang-test",
  "request": "launch",
  "name": "Debug pipeline tests",
  "mapping": "/abs/path/to/pipeline.yaml",        // launch target (.blobl or stream config)
  "testFile": "/abs/path/to/pipeline_bento_test.yaml",
  "testName": "",                                  // "" = all cases; a name = first match
  // "input": "/abs/path/input.jsonl",             // optional fallback input for cases without batches
  // "mockSteps": [2]                              // optional 1-based pipeline steps to mock
}

Extension Development Host smoke steps

npm install && npm run build
code --extensionDevelopmentPath="$PWD" "$PWD"   # opens an Extension Development Host window

In the Development Host window:

  1. Open the Testing view — the Bloblang Test Runner controller is present; run Bloblang Tests: Refresh and confirm the testdata/ definitions appear (6 files, including malformed_test.yaml with an error badge, and no pipeline_test.yaml/skipped_test.yaml).
  2. Run all on the workspace folder: passing cases go green, the known failures (dup#2, real jq fails, shouts failing, missing mapping) go red with detail in the Test Results panel.
  3. Debug a failing case (e.g. real jq fails): a bloblang-test debug session starts via blobl-dbg dap (needs blobl-dbg on PATH or blobltest.debugger.path set), output conditions are reported in the Debug Console, and the session exits.
  4. If the CLI binary is not on the EDH's PATH, set blobltest.runner.path to the absolute path of the built blobl-test-runner and refresh.

Layout

├── cmd/blobl-test-runner/   # Go CLI: list/run --json (engine-as-library)
├── src/                     # extension: extension.ts (explorer + profiles), cli.ts (CLI client)
├── scripts/                 # e2e.sh (CLI + DAP end-to-end), smoke-cli.mts (CLI-client smoke), dap-smoke.mts (DAP smoke)
├── testdata/                # fixtures incl. intentional failures
├── docs/adr/0001-runner-architecture.md
└── GLOSSARY.md
  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
  • Your Privacy Choices
  • Consumer Health Privacy
© 2026 Microsoft