DevAssure Invisible (QA) Agent

Extension for DevAssure Invisible (QA) Agent - Autonomous agent that can execute end to end UI tests from natural language instructions.

How it works

After you log in and initialize a project, the extension uses a .devassure folder in your workspace for configuration and test cases. You can add tests as YAML files or CSV, then run them from the DevAssure sidebar (Tests view) or from the editor via CodeLens. Execution logs stream in the Run Logs panel, and you can open reports from the History view or from archived zip files.

Execute tests from yaml and get intelligent reports

Setup Instructions

Login

1. In the left sidebar, click the DevAssure icon to open the DevAssure view, then click Login (shown in the welcome content when you are not logged in).
2. Your browser opens for OAuth2 authentication with DevAssure.
3. For CI or headless environments, you can add a token via the CLI: devassure add-token <your-token>.

Initialization

1. In the DevAssure left sidebar, click Initialize DevAssure (shown in the welcome view when no project is configured).
2. When prompted, choose the workspace folder to initialize.
3. Enter your app URL, app description, and one or more personas (name and short description).
4. The extension creates a .devassure folder with configuration files and a sample test. If .devassure already exists, you will be asked whether to override it.

The .devassure folder

The parent directory of .devassure is the project path. All DevAssure configuration and tests live under .devassure/:

Path	Purpose
app.yaml	App description and rules for the agent
personas.yaml	(Optional) User personas used in tests
test_data.yaml	Default URL, users, and test data per environment
agent_instructions.yaml	Instructions for the agent during execution
preferences.yaml	Browser and execution preferences (e.g. headless, workers)
csv_mapping.yaml	Column mappings when running tests from CSV
tests/ or test/	Directory for test case YAML files and optional CSV files
actions/	Directory for action YAML files (reusable step groups)

Configuration YAML files (overview)

• app.yaml – description (multi-line) and rules list for the agent.
• personas.yaml – Optional personas with description and optional fields (e.g. age_group, region).
• test_data.yaml – Top-level keys are environment names (e.g. default, uat). Each has url, users, and any custom test data. The extension’s Environment setting selects which key is used.
• agent_instructions.yaml – instructions list of strings for the agent (e.g. retry rules, fallbacks).
• preferences.yaml – browsers.default (e.g. browser: chromium, resolution, headless) and workers.
• csv_mapping.yaml – Maps CSV column names to test case fields: summary, steps, priority, tags (and optionally folder).

Sample configuration files

All configuration files live under .devassure/. Below are sample contents for each file type.

app.yaml – App description and rules for the agent:

description: >
  About my app
  in multi line
rules:
  - User can't create more than 3 projects
  - Any data deleted can't be restored

personas.yaml – (Optional) User personas used in tests:

normal_user:
  description: No admin access can do all other operations
  age_group: 18-30
  gender: M
  region: USA
admin:
  description: Admin can add or delete other users and manage privileges
deactivated_user:
  description: Login is deactivated by admin
expired_user:
  description: License has expired

test_data.yaml – Default URL, users, and test data per environment. If a key is not present for the selected environment, the default data is used:

default:
  url: 'http://localhost:3000'
  users:
    default:
      user_name: 'user@test.com'
      password: 12345678
    admin:
      user_name: 'admin@test.com'
      password: 12345678
  stripe_data:
    allowed_card:
      card: 4444 4444 4444 4444
    wrong_cvv:
      card: 1234 1234 1234 1234
      expiry: 10/28
      cvv: 123
  test_otp: 1122
uat:
  url: 'http://uat.myapp.com'
  users:
    default:
      user_name: 'user@uat.com'
      password: 12345678
    admin:
      user_name: 'admin@uat.com'
      password: 12345678
  test_otp: 2345

agent_instructions.yaml – Instructions for the agent during test execution:

instructions:
  - Reload the app and retry if app shows warning server is busy
  - Sign up a new user and proceed if any of the given logins are not working

preferences.yaml – Browser and execution preferences:

browsers:
  default:
    browser: chromium
    resolution: 1920 x 1200
    headless: true
workers: 2

csv_mapping.yaml – Column mappings when running tests from a CSV file. Maps CSV column names to test case fields:

summary: Summary
steps: Steps
priority: Priority
tags: Tags

Sample test case (e.g. .devassure/tests/sample-test.yaml):

summary: Sample test case
steps:
  - Open the application url
  - Verify if the page loads successfully
  - Verify if there are no error messages
priority: P1
tags:
  - sample
  - ui

Adding test cases

YAML test cases

Add .yaml or .yml files under .devassure/tests/ (or .devassure/test/). Each file should define at least summary and steps; priority and tags are optional.

Example:

summary: Sample test case
steps:
  - Open the application url
  - Verify if the page loads successfully
  - Verify if there are no error messages
priority: P0
tags:
  - sanity
  - app-load

From the extension

In the DevAssure sidebar, open the Tests view. Right-click the tests root (or any folder under it):

• Add Folder – Creates a new subfolder (e.g. my-suite).
• Add Test Case – Prompts for a name, creates a new .yaml file with placeholder content (summary, priority, tags, steps), and opens it in the editor.

Running tests

From the left panel (Tests view)

• Single file: Click the run (play) icon on a test file, or use the context menu Run Test. Only that file is executed.
• Folder or all tests: Click the run icon on a folder or on the tests root to run all tests in that scope. The saved filter (see Filters) is combined with the folder scope when running a folder.
• While a run is active, a Tests Running... item appears; click it to open the Run Logs panel.

From the editor (CodeLens)

• Open a test file under .devassure/tests/ (or test/). A CodeLens Run Test (DevAssure Agent) appears at the top; click it to run that file.
• Open a CSV file under .devassure/. A CodeLens Run Test Rows (DevAssure Agent) appears; click it to run tests from that CSV.

Run Logs panel

The DevAssure panel in the bottom panel area shows Run Logs – the live execution feed for the current session. Open it by clicking Tests Running... in the Tests view when a run is active, or by opening the bottom panel and selecting the DevAssure tab, then Run Logs.

Running tests from a CSV file

CSV mapping config (csv_mapping.yaml)

Create .devassure/csv_mapping.yaml to map your CSV column headers to test case fields. Supported keys: summary, steps, priority, tags (and optionally folder).

Example:

summary: Summary
steps: Steps
priority: Priority
tags: Tags

If your CSV uses different headers (e.g. from TestRail or Jira), set the right-hand side to the exact column name from your CSV. For example, if your export has columns "Title" and "Steps (Expected Result)", use summary: Title and steps: "Steps (Expected Result)" in csv_mapping.yaml.

Default mapping

If csv_mapping.yaml is missing, the agent may expect default column names such as Summary, Steps, Priority, Tags. For custom CSVs, add csv_mapping.yaml so columns map correctly.

From the sidebar

• Run a specific CSV: In the Tests view, CSV files under .devassure/tests/ (or test/) show a run icon. Click it to run that CSV (the saved filter applies to which rows are run).
• Run tests from CSV: Use the Run tests from CSV item at the bottom of the Tests view. A file picker opens (defaulting to .devassure); choose a CSV file to run.

From Editor (CodeLens)

Open a CSV file under .devassure/ in the editor. The CodeLens Run Test Rows (DevAssure Agent) runs that CSV with the current filter.

Configuration files

File	Purpose
app.yaml	App description and rules for the agent
personas.yaml	Optional user personas
test_data.yaml	URL, users, and test data per environment
agent_instructions.yaml	Agent instructions during execution
preferences.yaml	Browser (e.g. chromium), resolution, headless, workers
csv_mapping.yaml	CSV column → test field mapping
tests/ or test/	Test case YAML and optional CSV files

See Sample configuration files above for full YAML examples.

Filters

How to apply filters

In the DevAssure Tests view, click the Filter: … row (it shows the current filter or “nil”). An input box opens. Enter a filter string and press Enter to save.

Saving and clearing filters

• To save: Enter a valid filter and confirm. The filter is stored in workspace state and used for the next folder or CSV run.
• To clear: Open the filter input and submit an empty value (or clear the text and confirm).

Invalid syntax is rejected with an error and not saved.

Supported fields and operators

• Fields: tag, priority, folder, query
  • query performs a text match on summary, steps, and tags.
• Operators: = for field values; && and || between clauses; parentheses for grouping.

Examples:

• tag=smoke && priority=P0
• query=login
• (tag=user && priority=P0) || (tag=dashboard && priority=P1)

Where filters apply

Filters apply when running tests, not to the file tree:

• Folder run: Running a folder from the Tests view uses the folder scope combined with the saved filter.
• CSV run: Running a CSV file (from the sidebar or Editor) applies the saved filter to the rows in the CSV.

The Tests view always shows all files and folders; it is not filtered by the saved filter.

Reports / History

History view

Open the DevAssure sidebar and select History. The view lists the last 20 test sessions. Each entry shows the session title, ID, and timing.

Opening a report from history

Click a session in the History view. The extension opens that session’s report in your default browser. If no report server is running, it starts one and reuses it when possible.

View all sessions: Use the View all sessions item at the bottom to open the report server without a specific session.

Opening a report from archive

If you have an archived report zip (e.g. from devassure archive-report or devassure run-tests --archive=...):

1. In the History view title bar, click Open Report from Archive (archive icon).
2. Choose a .zip file in the file picker.
3. The report is served from the archive and opened in the browser (no database is used).

Actions

Actions are used to group steps that can be reused across multiple tests. To call an action you can simply mention the action name as a step in the test file.

Fields in action file

Field	Description
name	The name of the action. It is used to call the action from the test file.
description	A short description of the action.
steps	A list of steps that make up the action.

Example action file

Example 1 (e.g. .devassure/actions/login_as_admin.yaml):

name: login_as_admin
description: Login to the app as admin using google
steps:
  - Open admin portal url
  - Click on Google login button
  - Enter admin email and password
  - If MFA is asked, enter the authenticator OTP
  - If allow access is asked, click on allow access

Example 2 (e.g. .devassure/actions/add_manager_user.yaml):

name: add_manager_user
description: Add a manager user to the app
steps:
  - Go to users page
  - Click on Add User button
  - Enter email
  - Enter role as Manager
  - Once team dropdown appears assign any team to the user
  - Enter random user details
  - Click on Add User button

Usage in test file

In your test case, reference actions by name in the steps list:

steps:
  - login_as_admin
  - add_manager_user
  - Open users list page
  - Verify if the manager user is added

In the DevAssure sidebar, the Tests view shows an actions tree (when .devassure/actions exists). Right-click the actions root or a folder to Add Action (creates a new action YAML with name, description, steps template).

Advanced features

Switching environments

In the DevAssure Configuration view, click Environment: …. Enter an environment name (e.g. dev, uat, prod) that matches a top-level key in test_data.yaml. The selected environment is used for all subsequent test runs in that workspace.

Switching between multiple .devassure folders

If your workspace contains more than one project with a .devassure folder (e.g. multiple repos or packages):

1. Open the DevAssure Configuration view.
2. In the view title bar, click Select .devassure folder (folder icon).
3. In the picker, select either the .devassure folder or its parent directory. The path must be inside the current workspace.
4. The extension uses that project path for Configuration, Tests, and all runs until you change it again.

FAQ

How to add new test cases?
Add YAML files under .devassure/tests/ (or test/), or use the Tests view: right-click the tests root or a folder → Add Test Case or Add Folder. For CSV, add the CSV under .devassure/ (e.g. in tests/) and optionally configure .devassure/csv_mapping.yaml.

Is test case YAML always needed or can we use CSV only?
No. You can use CSV only: place the CSV in .devassure/ (e.g. under tests/), add csv_mapping.yaml if your columns don’t match the defaults, then run from the Tests view or via CodeLens on the CSV file.

What are the mandatory fields for a test case?
For YAML test cases, summary and steps are required. priority and tags are optional. For CSV, the mapped columns must provide at least the equivalent of summary and steps (via csv_mapping.yaml).

How to add a new environment?
Add a new top-level key (e.g. uat) and its data in .devassure/test_data.yaml. Then in the extension, set Environment (Configuration view) to that name (e.g. uat).

What are the supported browsers?
Controlled by .devassure/preferences.yaml under browsers.default.browser. The default is chromium. Other engines (e.g. firefox, webkit) may be supported depending on the agent; see preferences.yaml and the agent documentation.

How to run with headless mode?
In .devassure/preferences.yaml, set browsers.default.headless: true.

How to run the tests in Continuous Integration?
Use the CLI: authenticate with devassure add-token <token>, then run devassure run-tests with any options you need (e.g. --csv, --filter, --archive). The VS Code extension is intended for local and IDE use; CI should use the CLI.