BitAssert for VS Code

AppTestForce web test cases in your editor, authored with the AI CLI of your choice.

What it does

• Sign in to the platform. OAuth 2.0 authorization code + PKCE against AppTestForce-AS, with the organization (tenant) chosen the same way the web console chooses it. Tokens live in the OS keychain via vscode.secrets and refresh on their own. Every view reads one evaluated session, so the status bar and panels use the same connection state.
• Create and open web test cases. Organizations → projects → test cases in the activity bar. New test case writes a local draft and touches nothing else — the platform only hears about it when you publish it. Opening an existing one writes a readable JSON file and opens it.
• One project folder. Select your repository or requirements folder. Test cases, prompts, instructions and skills all live in its .bitassert/ folder, and selecting it creates the skill files straight away.
• Generate from two sources. Select a requirements document for one manual test case per scenario it describes, or ask the AI to inspect a repository feature. Each mode uses its own editable SKILL.md, with shared rules for platform-compatible output.
• Migrate existing UI tests. Pick a Selenium, Playwright or Cypress test and your AI CLI translates it into a reviewable draft with a migration report. See Migrating from another framework.
• Or keep them and expand around them. Expand coverage reads your existing tests as specifications and adds only what they leave unproven: negative cases, API edge cases and performance paths, starting from the routes no test reaches. See Existing tests as specifications.
• Give the AI your own instructions and skills. .bitassert/.instructions/ holds test-authoring rules, and .bitassert/.llm/ holds CLI guidance and skills. Regenerating preserves your edits.
• One file per test case, edited in place. The AI is given exactly one file to change — the one you have open — and is told not to create any. Save to platform publishes your reviewed work.
• Ask an AI in a terminal. Press Open AI terminal and you get a terminal in your project folder, pointed at your .instructions and .llm folders and at the one test case file it may edit. Requirements drafts also include their saved source text. The extension starts the CLI; type your request when it is ready. Works with Claude Code, Codex, Gemini, or any command you configure.
• Review before publishing. AI sessions enter edit mode before handing a file to the CLI. Ctrl+S keeps the draft local; Save to platform publishes the reviewed steps and details.
• Validation before you push. Action type names are checked against your project's own action types and locators against the platform's LocatorType values, as inline diagnostics.

Clicking the BitAssert icon in the activity bar (or running BitAssert: Open) opens the BitAssert page in an editor tab beside your code. Three tabs sit beside the logo: Home, Test cases and Review.

Home is laid out like VS Code's Get Started page. Set up walks through four steps — project folder, sign-in (browser or access token), AI CLI and skill files — with a check mark and progress bar as each is done, and opens the next one for you. Once all four are done, Set up folds into a row of chips (folder, AI CLI, skills) that reopen their step. Create holds Generate a test case (sources Document, Repository and Migrate tests) and Review & publish.

Test cases lists local drafts and your organizations' projects on the left; expand a project to load its web test cases. Clicking a test case shows, beside the list, its status, description, steps and actions (target element and value, without ids or locators); Open to edit opens its file, and Refine/Author opens it and has the AI refine it, giving manual steps actions and locators from the app's source.

Review is where generated test cases wait before anything reaches the platform: one row per test case, with its name, description, steps, preconditions, priority and status. Long cells clamp with Show more. Organization and Project above the table choose where they publish (the same working project as BitAssert: Select Project). Set a row's priority and status there, tick the rows to keep, and Publish; Delete removes rows from review. Every generated test case is published with the type Functional. View in the Steps column opens a row in a tab of its own, where Refine/Author has the AI edit that row in place, giving manual steps actions and locators from the app's source. Hover or focus an ⓘ icon for explanations. Signing out is in the page header; switching organization is a command-palette action.

There is no separate agent process. Everything runs from the editor: the platform over HTTPS, and the AI as a terminal you can see. ARCHITECTURE.md documents how each piece works.

Requirements

The defaults are the hosted BitAssert platform, so a fresh install needs no URL configuration.

Pointing at a different platform

When building the extension yourself, .env holds one block per environment (local, preprod, production). Uncomment one, then npm run build (F5 and vsce package run it too) bakes its BITASSERT_* URLs into dist/extension.js. .env is git-ignored and never packaged.

The URLs the build carries win over VS Code settings, so a leftover testcaseStudio.* URL in your user or workspace settings cannot point a build at the wrong platform. The settings apply only to a build made without .env values — a local stack, for example:

// .vscode/settings.json — read only when the build carries no BITASSERT_* URLs
{
  "testcaseStudio.platformUrl": "http://localhost:8090/api",
  "testcaseStudio.authUrl": "http://localhost:9000",
  "testcaseStudio.consoleUrl": "http://localhost:5173"
}

authUrl must be the same string as the issuer AS stamps into its tokens (jwt.issuer_url) and that RS validates against (spring.security.oauth2.resourceserver.jwt.issuer-uri) — behind a reverse proxy, that means the public URL rather than the internal one.

CORS is not involved on the extension's side: it is a Node HTTP client, not a browser. The console's CORS origins (ui.ip_port on AS and RS) now accept a comma-separated list, including wildcard hosts for preview deployments. ARCHITECTURE.md §12 has the full table of every place a URL lives, the certificate and proxy settings, and the one browser-side prerequisite for sign-in.

One-time setup on the authorization server

The extension signs in with its own public client, registered by BootstrapApplicationRunner in AppTestForce-AS:

clientId      vscode-extension-pkce
grant types   authorization_code, refresh_token
redirect      http://127.0.0.1:8899/callback
PKCE          required, no consent screen

The port in the redirect URI is a placeholder. The extension binds a free port each time it signs in, and Spring Authorization Server accepts any port on a loopback-IP redirect. Start AS once after pulling that change and the client is created automatically.

Sign-in goes through the web console

AppTestForce-AS has no sign-in page of its own. Its form login is disabled and its /oauth2/authorize endpoint requires an existing session, so a browser with no session there is turned away with a 401 rather than offered somewhere to log in.

So BitAssert: Sign In opens the console's sign-in page — testcaseStudio.consoleUrl + /login, by default https://app.preprod-bitassert.com/login — and not AS, carrying the authorization request it wants to make as a returnTo parameter. Signing in there leaves on the AS origin exactly the session cookie the authorization request needs, and the console then continues to that request in the same tab; AS answers it with a redirect to the extension's loopback listener, which is where the code was always going to arrive. One tab, one password, nothing to remember.

A console deployment that predates returnTo ignores it and stops at its own dashboard. Nothing is broken at that point — the browser is holding the session the flow was waiting for, one navigation short of finishing — so after thirty seconds the extension offers a Continue button that makes that navigation. If the wait goes quiet for some other reason, the progress notification says so after fifteen seconds and can be cancelled, and the failure offers to open the console and retry.

Getting started

1. Press Select project folder on the BitAssert page (or run BitAssert: Select Project Folder) and choose the application repository or requirements folder — an open root, a subfolder of one, or any other folder, which is opened in VS Code first. BitAssert writes the skill and instruction files into <folder>/.bitassert/ straight away. Setup shows the folder, the AI CLI and the skill files; Change there picks another folder.
2. BitAssert: Sign In — a browser opens on the BitAssert console's sign-in page; signing in there sends it on to the authorization request, and the code comes back to the extension. Pick your organization if the account has more than one. See Sign-in goes through the web console.
3. Run BitAssert: Select Project to choose the platform project. Confirm that your chosen AI CLI is installed and authenticated; the default is claude on PATH.
4. In Generate a test case, choose Document or Repository. To bring over existing Selenium, Playwright or Cypress tests instead, choose Migrate tests, or Expand coverage to keep them and add what they miss. Name the feature; for requirements, select the source file when prompted. The AI writes as many test cases as the request needs into one review file, a JSON array in .bitassert/review/. A document produces one manual test case per scenario, with a step for each of its cases (main path, alternatives, validation, boundaries). A step is a name with no actions: what the tester does and what they should then see. Publishing saves the steps on the platform and writes the step ids it gives them back into the review file.
5. When the CLI writes them, the page switches to Review. Choose the organization and project, check each row's priority and status, select the ones to keep and Publish. Each is created on the platform with its steps, and its row is locked. A row whose step save failed is marked Steps not saved and is retried, not created twice, on the next Publish.

The platform has no preconditions field, so a row's preconditions are added to the published description under a Preconditions: heading. Priority is matched by name to the platform's own; a value the platform does not have shows as (not on the platform) until you pick one. The type is always Functional; publishing stops with a message if the project has no type of that name. ✎ beside a name (on hover) renames a row that is not published yet; click the name itself to open the review file and edit anything else. Reopen unfinished work from Local drafts in the page's Test cases list or Open test case. Local drafts remain available after reloading VS Code and while signed out. Delete draft removes an unpublished case after confirmation. For an existing case, open it from the page's Test cases list and use Ask AI About This Test Case or Open AI Terminal Session. Both enter edit mode, so the same review step applies.

Requirements files must be UTF-8 plain text in .md, .markdown, .txt, .feature, .rst, or .adoc format, up to 256 KiB. Export Word or PDF requirements to one of those text formats first.

Giving the AI your own rules

Selecting the project folder creates these files; generation adds any that go missing, and BitAssert: Regenerate Skill Files refreshes them. BitAssert reads the folder's frameworks, scripts, ports and screens, then scaffolds <project folder>/.bitassert/:

.bitassert/
├── .instructions/
│   └── testcase-authoring.md     you write: what a good test case looks like here
├── .llm/
│   ├── claude.md                 you write: how Claude Code should go about it
│   ├── codex.md                  you write: how Codex should
│   ├── gemini.md                 you write: how Gemini should
│   ├── custom.md                 you write: how your own CLI should
│   ├── skills/natural-language/SKILL.md       editable: scenarios from prose
│   ├── skills/requirements-document/SKILL.md  editable: a manual test case per scenario
│   ├── skills/repository/SKILL.md             editable: flows grounded in source code
│   ├── skills/migration/SKILL.md              editable: translating tests from other frameworks
│   ├── skills/migration/FRAMEWORKS.md         editable: the translation table and your helper mappings
│   ├── skills/web-testcase/SKILL.md           editable: shared platform authoring rules
│   ├── skills/<name>/SKILL.md                 you write: additional team skills
│   ├── context/testcase-schema.md    generated: the file schema and this project's action types
│   ├── context/project-overview.md   generated: what the linked project is
│   └── config.json                   generated: what was detected, and when
├── testcases/
└── prompts/

The split is the point. .instructions is about the test case — locator conventions, naming, what must always be asserted — and applies whichever tool is running. .llm/<tool>.md is about the tool — how much it should explain, how it should use its own editing commands — and only the file for the CLI you have selected is ever read, so they can contradict each other freely.

testcase-authoring.md and the five built-in skills ship with usable guidance. Regeneration preserves these authored files, including your changes, and refreshes the derived overview, schema, configuration and README files. An AI request includes the selected generation mode's skill, shared web rules and custom skills; unrelated built-in modes are excluded.

Test cases and prompts are saved beside them, in .bitassert/testcases/ and .bitassert/prompts/.

Where they appear to disagree, .instructions wins over .llm/<tool>.md, and both win over the platform's project-wide instruction file. Turn the whole thing off with testcaseStudio.useLlmFolder.

Migrating from another framework

Migration is the Migrate tests tab of Generate a test case on the BitAssert page. Help › Welcome › Walkthroughs › Migrate UI tests to BitAssert walks through it step by step; Full guide in the step opens the same walkthrough.

1. Optionally type instructions for the AI (they go with every file), then press Migrate to search the project folder, or right-click a test folder in the Explorer and choose Migrate to BitAssert. A single file works too: Migrate <file> only when it is open in the editor, or right-click the file.
2. BitAssert lists the test files in the project folder (or the folder you right-clicked) and its subfolders (skipping node_modules, build output and dot folders) with every one ticked; untick any to leave out. Files that define no test, such as page objects and helpers, are not listed: the AI reads them while translating.
3. Each file becomes one test case with the same name as the file, without its extension (login.spec.ts → login.spec), holding every test in the file in order. Your AI CLI migrates the files one at a time, each into its own review file.
4. Read each migration report in the terminal: substitutions made, lines that could not be migrated, and any @{variables} to define on the platform. Restarting the AI session stops the migration; test cases already written stay, and the files not yet migrated are listed.
5. Review the test cases in the Review tab and Publish the ones to keep.

Tests under Jest and other runners that drive no browser have nothing to translate. They are left out of a folder migration, and still count as evidence in test-ontology.md. A file in which every test repeats another file's starts unticked. The test file is read from the open editor if it has unsaved changes, must be UTF-8 text up to 256 KiB, and is never modified.

The translation is your LLM's, following your instructions. Two editable files drive it:

• skills/migration/SKILL.md is the workflow: migrate a file's tests faithfully and in order, inline hooks, page objects and helpers, convert locators, replace sleeps with waits, move secrets to variables, never fake an unsupported action, and finish with a report.
• skills/migration/FRAMEWORKS.md is the translation table, intent by intent, from Selenium and Playwright to BitAssert action and locator types, with skills/migration/CYPRESS.md beside it for Cypress. Add your page objects and helpers under Team mappings (Framework table in the Migrate step) so every migration translates them the same way.
• skills/migration/API_SHORTCUTS.md says when and how setup steps run over HTTP instead of through the UI, using an API step group or a browser fetch in a JS Expression. A test keeps the form it exists to prove, and each shortcut is reported as a substitution.

Not migrated, and reported instead: network mocking, database setup, saved storage state, extra tabs and alerts, key presses, drag and drop, downloads and visual checks. A data-driven test is migrated with its first data row.

Existing tests as specifications

Whichever way you go, BitAssert first maps the tests already in your project into .bitassert/.llm/context/test-ontology.md, and refreshes it when a session starts if the copy is more than five minutes old:

• each test file, with the actions it exercises, the routes it targets and the state it verifies;
• cold spots, the routes your application declares that no test reaches, and the routes that are only ever mocked;
• API shortcut candidates, UI forms that only set the stage for a test;
• duplicate tests, with the same pages, actions and checks as an earlier one.

It is built by pattern matching, so the AI is told to confirm each entry in the file it names. Values typed into fields are never recorded, and literals on lines about credentials are redacted.

A test case cannot start an external script or mock the network; the AI reports those instead of faking them.

Working with the file

A test case is mirrored as JSON — not the platform's raw entity, so audit columns and lookup objects are gone and the screen name is flattened. It is meant to be read and edited by hand as well as by the AI. See examples/example-testcase.json:

{
  "$schema": "bitassert-web-testcase-v1",
  "id": "…", "name": "Login with valid credentials", "projectId": 1, "orgId": 1,
  "status": "DRAFT", "description": null, "baseUrl": null,
  "steps": [
    { "name": "Sign in", "enabled": true, "actions": [
      { "name": "Open the login page", "actionType": "Navigate To Url", "enabled": true,
        "input": "https://app.example.com/login", "element": null }
    ]}
  ]
}

A test case you create in the editor is written with these same keys — status, description and baseUrl are always present, null when you skipped them — plus a "draft": true marker that disappears after publishing to the platform. Only steps starts empty.

• actionType is the display name ("Click", "Enter", "Navigate To Url"), resolved against your project's action types on save. An unknown name stops the push and lists the valid ones.
• Ids identify existing rows. Keep them; removing a step or action from the file deletes it on the platform. The brief tells the AI the same thing.
• Typing in the file starts an edit. From then on nothing overwrites it — not opening the test case again, not pulling, not a refresh after someone else's save — and Ctrl+S saves to disk only. See Edit mode.
• After a successful push the file is rewritten from RS, so ids assigned to new steps, actions and elements land back in the file.
• Renaming works. A changed name, status, description or base URL goes to PUT /test_case/{id} before the steps are saved, because the step endpoint deliberately ignores those fields.
• Editing an element works, and changes it everywhere. Element names, screen names and locators go to PUT /element/{id}, because save_test_steps resolves an element by id and ignores the rest. Elements belong to the project, so BitAssert asks before writing them.

Edit mode

A mirror is in one of two states, shown in the Review & publish step and the status bar.

In sync — the file matches the platform. Saving it pushes it (testcaseStudio.autoPushOnSave), and BitAssert may refresh it from the platform whenever it has a newer copy.

Editing — you have changes the platform does not have. BitAssert will not touch the file:

• Ctrl+S saves to disk and nothing else. The status bar turns amber and reads not saved to platform; clicking it saves to the platform.
• Opening the test case again reveals the file rather than replacing it.
• Pull refuses, offering to save first or discard.
• Save to platform commits everything — elements, details, steps — then refreshes the file from the platform and returns you to in sync.
• Discard edits throws your changes away and takes the platform's copy. It asks first.

Editing starts on the first keystroke and before an AI request or terminal session is handed the selected case. BitAssert: Edit Test Case does the same thing explicitly.

What the platform will not let you change

save_test_steps writes a step's name, order and enabled flag, and nothing else. A step's waitTime, attribute or maxIterations cannot be changed from here — those are web-console fields. Rather than let the post-save refresh quietly revert them, BitAssert compares what came back against what it sent and tells you exactly which fields the platform kept:

Saved, but the platform kept its own values for step 1: the wait time is "1000" on the
platform, not "5000". Those edits are not in the file any more.

Action fields (name, type, input, output, enabled, waits) and everything on an element do save.

Commands

How an ask works

There are two ways to reach the CLI, and they hand it different things.

Open AI terminal prepares the project's context, hands the CLI paths, and starts it in the project folder. The CLI reads those instructions and waits for your request:

                    ┌── .bitassert/.instructions/*.md      what a good test case looks like here
                    ├── .bitassert/.llm/<tool>.md          how this CLI should go about it
<cli> "Read … " ────┼── .bitassert/.llm/context/           the file schema · this project's overview
   (cwd: your       ├── .bitassert/.llm/skills/*/SKILL.md  named know-how, when it matches
    project folder) └── .bitassert/testcases/<name>__<id>.json   the one file it may edit
   │
   ▼ file change detected
notification ──▶ Review / Save to platform ──▶ POST /test_step/save_test_steps

The extension submits the configured start command in a terminal in your project folder, for example:

claude "Read .bitassert/.instructions for how test cases are written here, and .bitassert/.llm for how to work — … You are working on the test case 'Checkout', which is the file .bitassert/testcases/checkout__1204.json — edit that file in place and no other … wait for my request."

Once the CLI is ready, type your request at its prompt. Opening the terminal does not submit a test-generation request on its own.

Interactive sessions point to the current context folders rather than copying their instructions into a brief. Opening a session clears older generated briefs. For a requirements draft, the extension also writes the original requirements snapshot to a source file, preserving the evidence used for generation even if the source document has since changed.

Generation mode, the original requirements snapshot and conversation are retained beside the prompt as .source.json and .history.json companions. Reopening a draft after reloading VS Code restores them; publishing carries them to the new platform identity. Draft conversations remain local, and new messages on published cases also sync with the platform.

Press the button again after selecting a different test case and the running session is told to move onto that one — the terminal outlives the selection, so this is what stops a session that was pointed at one test case from editing it when you meant another. The Review & publish step indicates when the two have drifted apart. The extension watches the whole mirror folder, so an edit made that way is reported and offered for review exactly the same — and an edit to a test case that is not the one selected is called out rather than passing unnoticed. With no test case selected the whole folder is in scope, so "fill in the one I just created" works.

BitAssert: Ask AI About This Test Case is the other path, for when you already know the request and want it sent in one go. It does write a brief — .bitassert/prompts/<name>.prompt.md, with the schema, the action types, your instructions, the skills, the conversation so far and your request — and starts the CLI with Read <…>.prompt.md and do what it says. The file is left on disk on purpose: it is the exact brief the CLI was given, so when a result is wrong you can see why.

Your prompt and a summary of what the CLI changed are written to the test case's conversation on the platform, so the thread in the web console stays in step with what you do here — and the next ask includes that history.

Settings

AI terminals use Windows PowerShell on Windows and /bin/sh on macOS/Linux so command arguments are escaped consistently. Custom templates use that shell's syntax. Executable paths and prompt values are passed literally, including spaces, quotes, dollar signs and backticks.

Troubleshooting

The instruction landed in a shell prompt, not in the CLI. Shell integration detects when the CLI exits and starts it again for the next ask. On hosts without shell integration, press Restart in Review & publish (under Activity) after exiting the CLI — the next ask starts a fresh session.

command not found. The CLI is not on the PATH the terminal uses. Put an absolute path in testcaseStudio.aiCommand.

"… is not an action type in this project." The name has to match a row in your project's action type lookup. Hover the squiggle, or run BitAssert: Validate Test Case for the full list.

Sign-in never returns. The browser must be able to reach 127.0.0.1 on the port in the redirect. A proxy configured for all traffic, including loopback, will break it.

Saving times out. The error names the call that hung, e.g. POST /test_case did not answer within 30s. That is RS accepting the request and then blocking on something it does synchronously inside it — the usual culprit is the notification publish, which does a blocking Kafka send per active project user and waits max.block.ms (60s by default) when the broker is unreachable. Check AppTestForce-RS.log for Topic ... not present in metadata after 60000 ms and fix KAFKA_BOOTSTRAP_SERVERS; raising testcaseStudio.requestTimeout only makes you wait longer for the same failure. The BitAssert output channel lists every request with its duration.

"New test case" says it cannot find a priority or type. Those are read from GET /test_case_priority and GET /test_case_type, the same endpoints the web console uses. If neither answers, the extension falls back to the values already in use on the project — so creating the first test case in the web console unblocks every one after it. The BitAssert output channel lists each path that was tried and what it answered.

The CLI is reading the wrong repository. It runs in the project folder shown in Setup. Press Change there (or run BitAssert: Select Project Folder) to pick another; a folder outside the current window is opened in VS Code. Old user-level path settings are ignored.

The AI ignores what I wrote in .instructions. A file that still contains only headings and <!-- --> hints is deliberately skipped, so make sure you have written something under a heading. Check testcaseStudio.useLlmFolder is on, and use Setup → Instructions to open the file from the project folder.

My .llm/claude.md rules are not being applied. Only the file for the currently selected provider is read. Switch provider in Setup and the brief switches with it — CLI rules opens the file for the selected CLI.

My new test case never appeared on the platform. Generated cases stay local until Save to platform. Drafts have "draft": true in the file and a local id; publishing asks for a priority and a type, creates the test case, and renames the file to the id the platform assigned.

My local drafts disappeared after upgrading. Earlier versions could save test cases outside the project folder (testcaseStudio.dataDirectory); that setting is now ignored and the files are still where it pointed. Move them into <project folder>/.bitassert/testcases/. Published test cases can simply be reopened from the platform.

Limitations

• Web test cases only. API and performance test cases are not mirrored, and New test case creates AUTOMATION_WEB ones.
• No run control. Executing a test case is still done from the console.
• Text requirements only. Word/PDF inputs need a text export, and each selected document is limited to 256 KiB.
• Saving a new test case needs a priority and a type. Both are read from the platform. If neither endpoint answers, the extension falls back to the values already in use on the project — which means the very first test case in a brand new project would have to be created in the web console. Until then your draft sits safely on disk.

Architecture

ARCHITECTURE.md covers the sign-in flow and why it works the way it does, the mirror format and its id semantics, the two endpoints a save goes through, the AI hand-off, the failure modes and how to extend any of it.

Development

npm install
npm run build     # esbuild bundle -> dist/extension.js
npm run compile   # tsc type check -> out/
npm test          # compile, then run the Node regression tests in tests/

Press F5 to launch an Extension Development Host.

License

MIT — see LICENSE.