DevAssure Invisible (QA) Agent

Extension for DevAssure Invisible (QA) Agent - executes end to end UI tests from natural language instructions and csv files.

How it works

After you log in and initialize a project, the extension uses a .devassure folder in your workspace for configuration and test cases. You can add tests as YAML files or CSV, then run them from the DevAssure sidebar (Tests view) or from the editor via CodeLens. Execution logs stream in the Run Logs panel, and you can open reports from the History view or from archived zip files.

How it works

Execute tests from yaml and get intelligent reports

Execution + Reports

Setup Instructions

In the left sidebar, click the DevAssure icon to open the DevAssure view, then click Login (shown in the welcome content when you are not logged in).
Your browser opens for OAuth2 authentication with DevAssure.
For CI or headless environments, you can add a token via the CLI: devassure add-token <your-token>.

Initialization

In the DevAssure left sidebar, click Initialize DevAssure (shown in the welcome view when no project is configured).
When prompted, choose the workspace folder to initialize.
Enter your app URL, app description, and one or more personas (name and short description).
The extension creates a .devassure folder with configuration files and a sample test. If .devassure already exists, you will be asked whether to override it.

The .devassure folder

The parent directory of .devassure is the project path. All DevAssure configuration and tests live under .devassure/:

Path	Purpose
`app.yaml`	App description and rules for the agent
`personas.yaml`	(Optional) User personas used in tests
`test_data.yaml`	Default URL, users, and test data per environment
`agent_instructions.yaml`	Instructions for the agent during execution
`preferences.yaml`	Browser and execution preferences (e.g. headless, workers)
`csv_mapping.yaml`	Column mappings when running tests from CSV
`tests/` or `test/`	Directory for test case YAML files and optional CSV files
`actions/`	Directory for action YAML files (reusable step groups)
`tools/`	Custom tools; `tools/index.yaml` defines tools and optional setup/settings

Configuration YAML files (overview)

app.yaml – description (multi-line) and rules list for the agent.
personas.yaml – Optional personas with description and optional fields (e.g. age_group, region).
test_data.yaml – Top-level keys are environment names (e.g. default, uat). Each has url, users, and any custom test data. The extension’s Environment setting selects which key is used.
agent_instructions.yaml – instructions list of strings for the agent (e.g. retry rules, fallbacks).
preferences.yaml – browsers.default (e.g. browser: chromium, resolution, headless) and workers.
csv_mapping.yaml – Maps CSV column names to test case fields: summary, steps, priority, tags (and optionally folder).

Sample configuration files

All configuration files live under .devassure/. Below are sample contents for each file type.

app.yaml – App description and rules for the agent:

description: >
  About my app
  in multi line
rules:
  - User can't create more than 3 projects
  - Any data deleted can't be restored

personas.yaml – (Optional) User personas used in tests:

normal_user:
  description: No admin access can do all other operations
  age_group: 18-30
  gender: M
  region: USA
admin:
  description: Admin can add or delete other users and manage privileges
deactivated_user:
  description: Login is deactivated by admin
expired_user:
  description: License has expired

test_data.yaml – Default URL, users, and test data per environment. If a key is not present for the selected environment, the default data is used:

default:
  url: 'http://localhost:3000'
  users:
    default:
      user_name: 'user@test.com'
      password: 12345678
    admin:
      user_name: 'admin@test.com'
      password: 12345678
  stripe_data:
    allowed_card:
      card: 4444 4444 4444 4444
    wrong_cvv:
      card: 1234 1234 1234 1234
      expiry: 10/28
      cvv: 123
  test_otp: 1122
uat:
  url: 'http://uat.myapp.com'
  users:
    default:
      user_name: 'user@uat.com'
      password: 12345678
    admin:
      user_name: 'admin@uat.com'
      password: 12345678
  test_otp: 2345

agent_instructions.yaml – Instructions for the agent during test execution:

instructions:
  - Reload the app and retry if app shows warning server is busy
  - Sign up a new user and proceed if any of the given logins are not working

preferences.yaml – Browser and execution preferences:

browsers:
  default:
    browser: chromium
    resolution: 1920 x 1200
    headless: true
workers: 2

csv_mapping.yaml – Column mappings when running tests from a CSV file. Maps CSV column names to test case fields:

summary: Summary
steps: Steps
priority: Priority
tags: Tags

Sample test case (e.g. .devassure/tests/sample-test.yaml):

summary: Sample test case
steps:
  - Open the application url
  - Verify if the page loads successfully
  - Verify if there are no error messages
priority: P1
tags:
  - sample
  - ui

Adding test cases

YAML test cases

Add .yaml or .yml files under .devassure/tests/ (or .devassure/test/). Each file should define at least summary and steps; priority and tags are optional.

Example:

summary: Sample test case
steps:
  - Open the application url
  - Verify if the page loads successfully
  - Verify if there are no error messages
priority: P0
tags:
  - sanity
  - app-load

From the extension

In the DevAssure sidebar, open the Tests view. Right-click the tests root (or any folder under it):

Add Folder – Creates a new subfolder (e.g. my-suite).
Add Test Case – Prompts for a name, creates a new .yaml file with placeholder content (summary, priority, tags, steps), and opens it in the editor.

Running tests

From the left panel (Tests view)

Single file: Click the run (play) icon on a test file, or use the context menu Run Test. Only that file is executed.
Folder or all tests: Click the run icon on a folder or on the tests root to run all tests in that scope. The saved filter (see Filters) is combined with the folder scope when running a folder.
While a run is active, a Tests Running... item appears; click it to open the Run Logs panel.

From the editor (CodeLens)

Open a test file under .devassure/tests/ (or test/). A CodeLens Run Test (DevAssure Agent) appears at the top; click it to run that file.
Open a CSV file under .devassure/. A CodeLens Run Test Rows (DevAssure Agent) appears; click it to run tests from that CSV.

Test branch and Test commit (Tests view)

In the Tests view, two items run a full test session focused on git changes (same idea as the CLI devassure test command):

Test branch

Where: Click Test branch in the Tests view (requires a git repo at the project path).
Flow: Two quick picks in order:
1. Test branch — Pick a branch from the list (all local and remote branch names from refs/heads and refs/remotes) or type a branch name. Default: confirm with Enter and no selection / empty input to use the current branch (placeholder: “Branch to test (press enter to use current branch)”).
2. Base branch — Same list and typing behavior. Default: Enter with no selection / empty input uses the repository default branch (placeholder: “Base branch (press enter to use repo default)”).
Result: Starts a session to test the code diff between source = chosen test branch (or current) and target = chosen base (or default).

Test commit

Where: Click Test commit in the Tests view.
Arguments: A single quick pick of the 20 most recent commits (short hash and subject). You can select a row or type a commit ID (full hash).
Default: Enter with no selection / empty input uses the current commit (HEAD) (placeholder: “Commit ID (press enter to use current commit)”).
Result: Starts a session for testing the code changes in the given commit.

If no DevAssure project is selected, execution is in progress, or setup is still running, these items may be disabled or show a wait message.

Run Logs panel

The DevAssure panel in the bottom panel area shows Run Logs – the live execution feed for the current session. Open it by clicking Tests Running... in the Tests view when a run is active, or by opening the bottom panel and selecting the DevAssure tab, then Run Logs.

Run Logs

Running tests from a CSV file

CSV mapping config (csv_mapping.yaml)

Create .devassure/csv_mapping.yaml to map your CSV column headers to test case fields. Supported keys: summary, steps, priority, tags (and optionally folder).

Example:

summary: Summary
steps: Steps
priority: Priority
tags: Tags

If your CSV uses different headers (e.g. from TestRail or Jira), set the right-hand side to the exact column name from your CSV. For example, if your export has columns "Title" and "Steps (Expected Result)", use summary: Title and steps: "Steps (Expected Result)" in csv_mapping.yaml.

Default mapping

If csv_mapping.yaml is missing, the agent may expect default column names such as Summary, Steps, Priority, Tags. For custom CSVs, add csv_mapping.yaml so columns map correctly.

Run a specific CSV: In the Tests view, CSV files under .devassure/tests/ (or test/) show a run icon. Click it to run that CSV (the saved filter applies to which rows are run).
Run tests from CSV: Use the Run tests from CSV item at the bottom of the Tests view. A file picker opens (defaulting to .devassure); choose a CSV file to run.

From Editor (CodeLens)

Open a CSV file under .devassure/ in the editor. The CodeLens Run Test Rows (DevAssure Agent) runs that CSV with the current filter.

Run Logs

Configuration files

File	Purpose
`app.yaml`	App description and rules for the agent
`personas.yaml`	Optional user personas
`test_data.yaml`	URL, users, and test data per environment
`agent_instructions.yaml`	Agent instructions during execution
`preferences.yaml`	Browser (e.g. chromium), resolution, headless, workers
`csv_mapping.yaml`	CSV column → test field mapping
`tests/` or `test/`	Test case YAML and optional CSV files

See Sample configuration files above for full YAML examples.

Filters

How to apply filters

In the DevAssure Tests view, click the Filter: … row (it shows the current filter or “nil”). An input box opens. Enter a filter string and press Enter to save.

Saving and clearing filters

To save: Enter a valid filter and confirm. The filter is stored in workspace state and used for the next folder or CSV run.
To clear: Open the filter input and submit an empty value (or clear the text and confirm).

Invalid syntax is rejected with an error and not saved.

Supported fields and operators

Fields: tag, priority, folder, query
- query performs a text match on summary, steps, and tags.
Operators: = for field values; && and || between clauses; parentheses for grouping.

Examples:

tag=smoke && priority=P0
query=login
(tag=user && priority=P0) || (tag=dashboard && priority=P1)

Where filters apply

Filters apply when running tests, not to the file tree:

Folder run: Running a folder from the Tests view uses the folder scope combined with the saved filter.
CSV run: Running a CSV file (from the sidebar or Editor) applies the saved filter to the rows in the CSV.

The Tests view always shows all files and folders; it is not filtered by the saved filter.

Reports / History

History view

Open the DevAssure sidebar and select History. The view lists the last 20 test sessions. Each entry shows the session title, ID, and timing.

Opening a report from history

Click a session in the History view. The extension opens that session’s report in your default browser. If no report server is running, it starts one and reuses it when possible.

View all sessions: Use the View all sessions item at the bottom to open the report server without a specific session.

Resuming a test session

You can resume a previously run test session to continue execution from where it left off:

In the DevAssure sidebar, open the History view.
Right-click a session entry and choose Resume (or use the resume action from the context menu).
The extension starts the agent in resume mode for that session; the Run Logs panel shows the execution feed.

Resume is only available for sessions that support resumption and when no other run is in progress.

Archiving a report

To save a session's report as a zip file (e.g. for sharing or offline viewing):

In the DevAssure History view, right-click the session you want to archive.
Choose Archive report (or the archive action from the context menu).
When prompted, select the output folder for the archived report.
The extension creates devassure-results-<session-id>.zip in that folder. You can use Open Report from Archive to view it later.

Opening a report from archive

If you have an archived report zip (e.g. from devassure archive-report or devassure run-tests --archive=...):

In the History view title bar, click Open Report from Archive (archive icon).
Choose a .zip file in the file picker.
The report is served from the archive and opened in the browser (no database is used).

Actions

Actions are used to group steps that can be reused across multiple tests. To call an action you can simply mention the action name as a step in the test file.

Fields in action file

Field	Description
`name`	The name of the action. It is used to call the action from the test file.
`description`	A short description of the action.
`steps`	A list of steps that make up the action.

Example action file

Example 1 (e.g. .devassure/actions/login_as_admin.yaml):

name: login_as_admin
description: Login to the app as admin using google
steps:
  - Open admin portal url
  - Click on Google login button
  - Enter admin email and password
  - If MFA is asked, enter the authenticator OTP
  - If allow access is asked, click on allow access

Example 2 (e.g. .devassure/actions/add_manager_user.yaml):

name: add_manager_user
description: Add a manager user to the app
steps:
  - Go to users page
  - Click on Add User button
  - Enter email
  - Enter role as Manager
  - Once team dropdown appears assign any team to the user
  - Enter random user details
  - Click on Add User button

Usage in test file

In your test case, reference actions by name in the steps list:

steps:
  - login_as_admin
  - add_manager_user
  - Open users list page
  - Verify if the manager user is added

In the DevAssure sidebar, the Tests view shows an actions tree (when .devassure/actions exists). Right-click the actions root or a folder to Add Action (creates a new action YAML with name, description, steps template).

Library tools

Library tools are inbuilt tools that users can include in the project by listing the required tools in a library.yaml file.

Sample library.yaml (e.g. .devassure/library.yaml):

tools:
  - 'authenticator'
  - 'faker:*'

faker

Faker provides fake data generators for tests:

Tool	Description
`first_name`	Generate a random first name
`last_name`	Generate a random last name
`full_name`	Generate a random full name
`email`	Generate a random email address
`phone`	Generate a random phone number

authenticator

Authenticator provides TOTP (Time-based One-Time Password) helpers:

Tool	Description
`get_authenticator_otp`	Generate a TOTP code from an authenticator secret. The authenticator secret must be passed to generate the code.

Tools

In the DevAssure Tests view, a Tools tree is shown when .devassure/tools/index.yaml exists; you can open index.yaml or reveal the tools folder from there.

Tools let you run custom commands or programs and consume their output. The configuration lives in .devassure/tools/index.yaml.

Requirement: tools/index.yaml is required for tools to work. It must exist at .devassure/tools/index.yaml (relative to your project path).

Tools can run any command or program; the runner executes the command and consumes its output. For custom code or scripts, put the tool code inside the .devassure/tools folder and reference the execution command in tools/index.yaml (e.g. exec: node script.js or exec: npm run myTool).

Note: You can use any programming language or program. You are responsible for setting up dependencies (e.g. Node.js, Python, venv, system binaries).

Mandatory fields

Each tool must have name, description, and exec:

Field	Description
name	Unique identifier for the tool (e.g. used when invoking the tool).
description	Short description of what the tool does.
exec	Command(s) to run. Can be a single line or a multi-line string. Supports `${argName}` substitution for args.

Optional per-tool fields include cwd and args (see below).

Minimal example:

tools:
  - name: "getProjectDetails"
    description: "Get project details from api"
    cwd: "api-tools"
    args:
      - name: projectId
        type: string
    exec: npm run getProjectDetails ${projectId}

cwd (optional): Working directory for the command. It is relative to the tools folder (.devassure/tools). If omitted, the working directory is the tools folder. Absolute paths are also supported.
args (optional): List of parameters. Each has name and type; see Supported arg types and optional args below.

Optional top-level configuration: settings and setup

settings: Default options applied to every tool run and every setup step. For all fields except env, the value on the tool or setup step is used when present; values in settings are fallback when the tool or step does not define that field. For env, the list of environment variables is merged: settings env vars and the tool/setup step env vars are combined (tool/setup entries override settings when the same key is used).

setup: Optional list of steps run once per session before scenario executions start (e.g. install dependencies). Each step can specify name, cwd, exec, and any of the settings fields; inheritance from settings applies when a step omits a field.

cwd behavior

Relative: Resolved from the tools folder (e.g. cwd: "api-tools" → .devassure/tools/api-tools).
Default: If cwd is not provided, the working directory is the tools folder.
Absolute: Absolute paths are supported (e.g. cwd: "/opt/scripts").

Settings and tool/setup options

Option	Applies to	Description
timeoutSec	Tool / setup	Maximum execution time in seconds (e.g. `10`).
output.start / output.end	Tool / setup	Markers in stdout; only content between these markers is captured as the tool output. Strongly recommended when the process prints a lot of extra content (logs, progress). Your script should print the actual result between these markers.
env	Tool / setup	List of `KEY: value` environment variables added or overridden for the process.
ignore_failure	Tool / setup	If `true`, a non-zero exit or failure does not fail the run (e.g. optional setup). Default is `false`.

Output start and end markers

When the command prints a lot of unwanted content (logs, progress, debug), use output.start and output.end in settings (or on the tool). The runner captures only the stdout between these two markers. Print the markers and the necessary output from your script.

JavaScript example: print the markers, then the result, then the end marker:

// In your script (e.g. getProjectDetails.js)
const startMarker = "__TOOL_OUTPUT_START__";
const endMarker = "__TOOL_OUTPUT_END__";

// ... do work, then:
console.log(startMarker);
console.log(JSON.stringify(result));
console.log(endMarker);

In tools/index.yaml you would set output.start and output.end to match (e.g. __TOOL_OUTPUT_START__ and __TOOL_OUTPUT_END__).

Supported arg types and optional args

Supported types: string, number, boolean, object. Use type in each arg.

Optional args: Add optional: true to an arg. If not provided, it may be omitted or passed empty depending on how you use it in exec (e.g. "${projectName}" for an optional string).

Example with multiple types and one optional arg:

tools:
  - name: "exampleTool"
    description: "Example with string, number, boolean, object args"
    args:
      - name: id
        type: string
      - name: count
        type: number
      - name: verbose
        type: boolean
      - name: options
        type: object
      - name: projectName
        type: string
        optional: true
    exec: node run.js ${id} ${count} ${verbose} "${options}" "${projectName}"

exec format

Single line: exec: npm run getProjectDetails ${projectId}
Multi-line: Use YAML multi-line (e.g. exec: | with following lines). Useful for running several commands in sequence (e.g. warmup, main command, cleanup):

exec: |
  npm run warmupTestingProcess
  npm run getProjectDetails ${projectId} "${projectName}"
  npm run cleanupTestingProcess

Full example (tools/index.yaml)

The following example shows settings (timeoutSec, output markers, env, ignore_failure), setup (one step with cwd and exec), and tools with optional arg and multi-line exec:

settings:
  timeoutSec: 10
  output:
    start: "__TOOL_OUTPUT_START__"
    end: "__TOOL_OUTPUT_END__"
  env:
    - BUILD_ENV: "dev"
    - DB_NAME: "legacy-db"
  ignore_failure: false
setup:
  - name: "Install dependencies"
    cwd: "api-tools"
    exec: "npm install"
tools:
  - name: "getProjectDetails"
    description: "Get project details from api"
    cwd: "api-tools"
    args:
      - name: projectId
        type: string
      - name: projectName
        type: string
        optional: true
    exec: |
      npm run warmupTestingProcess
      npm run getProjectDetails ${projectId} "${projectName}"
      npm run cleanupTestingProcess

Advanced features

Switching environments

In the DevAssure Configuration view, click Environment: …. Enter an environment name (e.g. dev, uat, prod) that matches a top-level key in test_data.yaml. The selected environment is used for all subsequent test runs in that workspace.

Switching between multiple .devassure folders

If your workspace contains more than one project with a .devassure folder (e.g. multiple repos or packages):

Open the DevAssure Configuration view.
In the view title bar, click Select .devassure folder (folder icon).
In the picker, select either the .devassure folder or its parent directory. The path must be inside the current workspace.
The extension uses that project path for Configuration, Tests, and all runs until you change it again.

Add auth token

Instead of logging in via the browser, you can add an authentication token. Generate a token from app.devassure.io and run the DevAssure: Add Auth Token command (Command Palette or from the DevAssure view) to add it. The extension will validate and store the token so you can use DevAssure without signing in through the browser. This is useful for CI, headless environments, or when you prefer token-based authentication.

Advanced test_data.

You can keep large or reusable test_data in separate YAML files and merge them into your scenario config using !include, so teams can share the same data across multiple scenarios/environments. This works for both single values (strings/numbers/booleans) and arrays of values (e.g., multiple cards, multiple admin users, multiple payment methods).

Examples

Reusable actions YAML (multiple actions in one file)

# .devassure/actions/cart.yaml
- name: go_to_carts_page
  description: Go to the carts page and verify the content
  steps:
    - Click on the carts icon
    - Verify the carts page is opened

- name: add_to_cart
  description: Add a product to the cart
  steps:
    - Click on the add to cart button
    - Verify the product is added to the cart

Reusable actions YAML (single action in one file)

# .devassure/actions/products.yaml
name: remove_from_cart
description: Remove a product from the cart
steps:
  - Click on the remove button for the product
  - Verify the product is removed from the cart

Include reusable test_data into your test_data config

# .devassure/test_data.yaml
default:
  url: http://127.0.0.1:4321

  cart:
    <<: !include test_data/cart.yaml

Proxy

If the HTTP_PROXY or HTTPS_PROXY environment variables are set, the extension will use them to proxy requests. You can also set the proxy in the extension settings: DevAssure: Http Proxy (devassure.http_proxy) and DevAssure: Https Proxy (devassure.https_proxy). Settings take precedence over environment variables when both are present.

FAQ

How to add new test cases?
Add YAML files under .devassure/tests/ (or test/), or use the Tests view: right-click the tests root or a folder → Add Test Case or Add Folder. For CSV, add the CSV under .devassure/ (e.g. in tests/) and optionally configure .devassure/csv_mapping.yaml.

Is test case YAML always needed or can we use CSV only?
No. You can use CSV only: place the CSV in .devassure/ (e.g. under tests/), add csv_mapping.yaml if your columns don’t match the defaults, then run from the Tests view or via CodeLens on the CSV file.

What are the mandatory fields for a test case?
For YAML test cases, summary and steps are required. priority and tags are optional. For CSV, the mapped columns must provide at least the equivalent of summary and steps (via csv_mapping.yaml).

How to add a new environment?
Add a new top-level key (e.g. uat) and its data in .devassure/test_data.yaml. Then in the extension, set Environment (Configuration view) to that name (e.g. uat).

What are the supported browsers?
Controlled by .devassure/preferences.yaml under browsers.default.browser. The default is chromium. Other engines (e.g. firefox, webkit) may be supported depending on the agent; see preferences.yaml and the agent documentation.

How to run with headless mode?
In .devassure/preferences.yaml, set browsers.default.headless: true.

How to run the tests in Continuous Integration?
Use the CLI: authenticate with devassure add-token <token>, then run devassure run-tests with any options you need (e.g. --csv, --filter, --archive). The VS Code extension is intended for local and IDE use; CI should use the CLI.

DevAssure Invisible (QA) Agent

DevAssure

DevAssure Invisible (QA) Agent

How it works

Execute tests from yaml and get intelligent reports

Setup Instructions

Login

Initialization

The .devassure folder

Configuration YAML files (overview)

Sample configuration files

Adding test cases

YAML test cases

From the extension

Running tests

From the left panel (Tests view)

From the editor (CodeLens)

Test branch and Test commit (Tests view)

Test branch

Test commit

Run Logs panel

Running tests from a CSV file

CSV mapping config (csv_mapping.yaml)

Default mapping

From the sidebar

From Editor (CodeLens)

Configuration files

Filters

How to apply filters

Saving and clearing filters

Supported fields and operators

Where filters apply

Reports / History

History view

Opening a report from history

Resuming a test session

Archiving a report

Opening a report from archive

Actions

Fields in action file

Example action file

Usage in test file

Library tools

faker

authenticator

Tools

Mandatory fields

Optional top-level configuration: settings and setup

cwd behavior

Settings and tool/setup options

Output start and end markers

Supported arg types and optional args

exec format

Full example (tools/index.yaml)

Advanced features

Switching environments

Switching between multiple .devassure folders

Add auth token

Advanced test_data.

Examples

Reusable actions YAML (multiple actions in one file)

Reusable actions YAML (single action in one file)

Include reusable test_data into your test_data config

Proxy

FAQ