TestsExplorer

how to build

npm run install npm run compile npm run package

usefull links

codicons

pre paths

A root folder rarely holds its reports right away : they usually sit a few folders below, in a layout shared by every build. pre_paths describes that layout once, and every root folder of that shape follows it :

    "xml-tests-explorer.pre_paths": [
        {
            "category": "MR",
            "show_category": false,
            "name": "(MR_\\d\\d\\d\\d)",
            "pre_paths": [
                { "name": "pbr123", "path": "BASE/prj1" },
                { "name": "pbr456", "path": "BASE/prj2" },
                { "path": "BASE/prj4" }
            ]
        },
        {
            "category": "Dev",
            "name": "(.*)",
            "pre_paths": ["."]
        }
    ]

A folder is recognized by the first entry whose name matches its own name, and which holds at least one of its pre_paths. name is a regex on the name of the folder, never on a path, and what it captures becomes the label, prefixed by category between parenthesis unless show_category says otherwise. The pre_paths are plain paths, never regex : "." is the folder itself, and a precise path is what keeps the scan short, since the walk starts there instead of at the top.

name only sees the name of the folder, so two folders of a same name living in different places cannot be told apart by it. path is a regex on the absolute path of the folder, with / as separator whatever the system, matched as a whole : when it is given, the folder has to match both. It lets a category follow where the builds live, for instance one per user :

    "xml-tests-explorer.pre_paths": [
        {
            "category": "John",
            "name": "(MR_\\d\\d\\d\\d)",
            "path": ".*/users/1234/.*",
            "pre_paths": ["BASE/prj1"]
        },
        {
            "category": "Jane",
            "name": "(MR_\\d\\d\\d\\d)",
            "path": ".*/users/5678/.*",
            "pre_paths": ["BASE/prj1"]
        }
    ]

Each pre path is shown as an item of its own, holding the suites found in it, and named by its name — the name of the folder it points at by default. A pre path written as a plain string shows no item at all and puts its suites right below the folder, which is what "." does too.

An entry describes a shape as soon as one of its pre paths is a real path. An entry holding only "." describes nothing — "." is always there — it only says that the folder being pointed at is the one to explore. That difference drives what a root folder holds, in that order :

1. a described shape recognizes the root folder, it is the one explored
2. otherwise, the same shapes are looked for in its direct sub folders, one level and no deeper
3. otherwise, the root folder is taken as it is, by any entry, the catch all included

Several recognized sub folders make one item each, a single one is flattened in the root folder, and a root folder where nothing is recognized stays in the tree, empty, saying so in its tooltip. A recognized sub folder holding no test at all is not shown.

The order matters : a catch all entry recognizes every folder, so taking the root folder on it first would keep the projects below it from ever being looked for. None of this reads a single report, what a root folder holds is a matter of folders.

Leaving pre_paths out means [{ "name": "(.*)", "pre_paths": ["."] }] : every root folder is explored as it is.

what is going on

The tests explorer never explores anything on its own : opening vscode shows the line of every configured root folder, saying not explored yet, and walks nothing. Writing a root folder in the settings walks nothing either. A root folder is explored from the refresh icon of its own line, and it alone, so fifteen root folders never cost fifteen walks. The one exception is the Add tests root folder button, which explores what it just added : picking a folder there is asking for its tests. While one is being explored its line says scanning..., and the status bar holds an item naming what is being explored. Clicking that item offers to give up an exploration.

An exploration given up leaves its root folder with the tree it held, and records nothing of the run it did not finish.

A root folder that found no test at all stays in the tree, with a plain folder icon and no color : nothing was found, so nothing is said. Only a root folder that could not be found on the disk is red.

What each root folder held at its last exploration is kept between two sessions, so opening vscode brings back the whole tree — counters, colors and marks — without walking anything. Every exploration is dated, so hovering a root folder says when the tree you are looking at was made, down to the second : counters of five minutes ago and counters of last week do not say the same thing. The line itself stays on its counters.

grouping the root folders

group_root_folders_by_category, on by default, gathers the root folders sharing the category of the pre_paths entry that recognized them :

    "xml-tests-explorer.group_root_folders_by_category": false

MR                    0/2 OK's (24/30)
  1234
  5678
(JK) one
nothing_known

The category is not repeated in front of its folders : that prefix was the flattened form of the category line, so show_category only has a word to say when the grouping is off.

A category naming a single root folder gets no line of its own — it would say nothing that folder does not say. JK above names only one, so it stays where it is and keeps saying its category itself.

A category shows up where its first folder was, so the order of the settings is kept, and a root folder belonging to no category stays where it is. Its line carries a refresh icon of its own, exploring every folder it holds, one after the other. Finding out which category a folder belongs to costs no exploration : a folder is recognized on its name and on the presence of its pre paths, which is a handful of file checks.

root folders

The tests explorer shows one top level item per root folder. The root folders are stored in the user settings, whatever workspace is opened, and can be added or removed from the view title bar, or removed from the icons of the top level item itself. Each top level item also carries a refresh icon, exploring that root folder alone : nothing else is walked again, and removing or renaming one costs no walk at all. The edit icon of the same line asks for the folder itself in a file dialog, so that it does not have to be written in the settings file. The label is left as it is : it comes from the settings, written there by hand or derived by the pre paths. Adding a root folder opens the dialog on the folder last picked. The refresh icon of the view title bar reads the settings again, so that a change of the pre_paths shows at once in the categories and the labels, without walking any root folder :

    "xml-tests-explorer.test_roots": [
        {
            "name": "Nightly",
            "path": "D:/builds/nightly"
        },
        {
            "name": "Local",
            "path": "."
        }
    ]

name is optional and defaults to the folder name. path is absolute or relative to the workspace folder.

Two root folders carrying the same folder name take as much of their path as it takes to tell them apart, so two tests folders show up as nightly/tests and release/tests. A name written by hand is never touched. Whatever the line, the folder it stands for is in its tooltip, as is the file a test was read from.

order of the cases

The cases of a file come in their natural order : the order they were run in. A report such as the CppUnit one splits the failing cases from the other ones, but it numbers each of them, and xml_path_case_order tells where that number is read. An attribute is read with a @_ prefix, so <Test id='1'> gives :

    "xml_path_case_order": "@_id"

A suite giving no xml_path_case_order shows its cases by label, case2 coming before case10.

The title bar of the view switches between that natural order and the failing cases first, still in their natural order between them. The choice is the failed_first setting, kept from a session to the next, and switching costs no walk :

    "xml-tests-explorer.failed_first": true

dates of the tests

A report rarely says when its tests ran, so the date of a test is taken, in that order :

1. the date of the case, read below it at xml_path_case_date in a case of the suite
2. the date of the whole report, read at xml_path_date in the suite
3. the date the report file was written, which is when the run ended

    "xml-tests-explorer.test_suites": [
        {
            "category": "TU",
            "name": "TU/(.*)/Result.xml",
            "xml_path_date": "testsuites.testsuite.@_timestamp",
            "cases": [
                { "type": "failing", "xml_path_case": "...", "xml_path_case_date": "@_time" }
            ]
        }
    ]

An attribute is read with a @_ prefix. A date is an iso or a readable date, or a number of seconds or of milliseconds since 1970. The tooltip of a file or of a case says when it ran, and whether that is only the date of its report file.

Every date, in the tree, its tooltips and the report, is shown day first, as day/month/year hours:minutes:seconds, whatever the locale of the machine.

detailed report

Every line of the view carries a report icon : a category, a root folder, a pre path, a suite, a file, down to a single case. It opens a page summing up the tests below that line, one page per line, rendered again each time its root folder is explored again :

• the score, the failed cases, the files passing and the evolution (new errors, fixed, new) at the top, in the colors of the tests explorer
• the tallies by type (the category of the suites), by failure type and by file, the score of each group coming first
• the failing tests, under a section per type when there are several, with their failure type, message and source, and the tests fixed since the previous runs
• when the tests ran : a Time column for the cases and the files, the last run of each type, and the oldest and the latest runs at the top. A time in italic, marked (file) in markdown, is the date of the report file, the report giving none

Each section folds, and Fold all / Unfold all do it for all of them. The search field keeps the cases holding every word typed, in their name, type, file, message, source, or failed / passed, and every part of the page follows it.

A cell holds a single line, its whole text in its tooltip, and a click on a row unfolds it. A column header sorts the table on that column, after the kind of value it holds : numbers and scores as numbers, dates in time, texts by name with the numbers in them compared as numbers. A second click sorts down, a third one gives back the order of the report. Dragging the edge of a header resizes the column, and Columns shows or hides them. A table wider than the page scrolls in its own box. All of that is kept from a search to the next, for every table of the same kind.

A click on a source opens it at its line, a click on a file opens its xml report. Copy as Markdown copies the report, the search applied, and the buttons of a failing test copy its message alone (msg) or the whole case as markdown (case).

Export HTML writes the report as a single html file : its data, its style and its script are inside, so it opens in any browser, from the disk, without vscode, network or library. It does everything the page of vscode does, the search, the sort, the folding, the columns, the copies, in the light or dark theme of the system. Its links open the sources and the xml reports as file:// urls, and it says when it was exported : it is a snapshot, it does not follow the next explorations. What it is set to, the search, the sort, the columns, is kept in the storage of the browser when the browser gives one to a file opened from the disk, otherwise it starts over each time.

a complete working example

The samples folder of this repository holds real reports, laid out like this :

samples/Project/Tests/Module1/TU/Result_CPPUNIT.xml
samples/Project/Tests/Module2/TU/Result_CPPUNIT.xml
samples/Project/Tests/Module3/TU/Result_CPPUNIT.xml
samples/Project/Results/TOTO_0045.xml
samples/Project/Results/TOTO_4577.xml

Pointing a root folder at samples/Project with the settings below gives :

Project            0/2 OK's (32/37)
  TU               1/3 OK's (32/34)
    Module1        16/16 OK's
      TOTO::Test1
      TATA::Test1
      ...
    Module2        16/17 OK's
    Module3        0/1 OK's
  LUA              0/2 OK's (0/3)
    TOTO_0045      0/2 OK's
      TEST_0045::Fichier_XML_resultats_non_produit
    TOTO_4577      0/1 OK's
      TOTO::Ce test est merdique

Every line counts twice : its direct children on the left, the whole subtree it holds between parenthesis. TU holds 3 modules of which 1 is fully green, and 34 cases of which 32 pass.

Had that root folder held TU alone, the line would read Project / TU and the modules would sit right below it : a level holding one thing says nothing more than that thing, so it is not given a line of its own.

The two categories read the same kind of report, they only differ by where the files are and by what names them : TU takes the module name out of the path, LUA takes the file name. The name of a suite is a regex on the path of the file relative to the root folder, and its capture is the label of the item holding the cases.

    "xml-tests-explorer.test_roots": [
        {
            "name": "Project",
            "path": "D:/dev/XmlTestsExplorer/samples/Project"
        }
    ],
    "xml-tests-explorer.test_suites": [
        {
            "category": "TU",
            "name": "Tests/(.*)/TU/Result_CPPUNIT.xml",
            "base_path": "Tests/",
            "cases": [
                {
                    "type": "failing",
                    "xml_path_case": "TestRun.FailedTests.FailedTest",
                    "xml_path_case_label": "Name",
                    "xml_path_case_source_file": "File",
                    "xml_path_case_source_line": "Line",
                    "xml_path_case_message_type": "FailureType",
                    "xml_path_case_message_content": "Message"
                },
                {
                    "type": "success",
                    "xml_path_case": "TestRun.SuccessfulTests.Test",
                    "xml_path_case_label": "Name",
                    "xml_path_case_source_file": "File",
                    "xml_path_case_source_line": "Line",
                    "xml_path_case_message_type": "FailureType",
                    "xml_path_case_message_content": "Message"
                }
            ]
        },
        {
            "category": "LUA",
            "name": "Results/(.*).xml",
            "base_path": "Results/",
            "cases": [
                {
                    "type": "failing",
                    "xml_path_case": "TestRun.FailedTests.FailedTest",
                    "xml_path_case_label": "Name",
                    "xml_path_case_source_file": "File",
                    "xml_path_case_source_line": "Line",
                    "xml_path_case_message_type": "FailureType",
                    "xml_path_case_message_content": "Message"
                },
                {
                    "type": "success",
                    "xml_path_case": "TestRun.SuccessfulTests.Test",
                    "xml_path_case_label": "Name",
                    "xml_path_case_source_file": "File",
                    "xml_path_case_source_line": "Line",
                    "xml_path_case_message_type": "FailureType",
                    "xml_path_case_message_content": "Message"
                }
            ]
        }
    ]

No pre_paths here : the root folder holds Tests/ and Results/ right away, which is what the default [{ "name": "(.*)", "pre_paths": ["."] }] means. Add pre_paths only when the reports sit deeper, or when you would rather point at a folder holding several projects.

Enjoy!