ETL Explorer

Explore ETL items in VS Code. An ETL item is a *.etl folder holding an npy-store-v1 columnar store written by etl-agent. ETL Explorer reads the manifest and every NumPy file in the item and shows:

• an overview with a coverage map
• signals in interactive multi-lane plots
• FFT columns as spectrograms
• statistics, correlation, raw tables and file headers

Everything runs locally and the extension has no runtime dependencies.

Opening an item

• In the Explorer, right-click a *.etl folder and choose Open in ETL Explorer.
• Or click manifest.json inside an .etl folder. ETL Explorer is the default editor for that file. Use Open Manifest as Text in the title bar to see the JSON.
• Or click any .npy file. If the file belongs to a column of an ETL item, the whole item opens with that column plotted. A loose .npy opens on its own: 1-D arrays as a plot, 2-D arrays as a heatmap.
• The ETL Items view in the Explorer lists every *.etl folder in the workspace and its columns. Click an item to open it, or a column to plot it. The view searches the workspace once when it is first shown. Press Refresh to search again. There are no file watchers and no polling.
• ETL Explorer: Open ETL Item… asks for a folder. Folders without a manifest.json work too: the layout is then inferred from the file names.

What an ETL item contains

Signal names that contain /, such as Motor Current/Torque Fbk, are stored in nested folders. The signal list shows them under their group.

Views

Overview

• Item details from the manifest (timestamps are also shown as dates), sources, notes, and the gaps between windows.
• A coverage map: one row per column, one cell per window. Green means new data, amber means held, gray means no data. Hover for details; click a row to plot that column.

Plot

• Click signals in the list to add them. Shift+click a group to add all of its signals.
• Lanes gives each signal its own lane and y-axis, with a shared time axis. Overlay puts signals on one axis. You can normalize them to 0–1 or to z-scores so that signals in different units can be compared.
• Held windows can be shaded (the default), hidden, or drawn plain. Gaps between windows are shaded darker.
• Drag to zoom. Use the wheel to zoom when the lanes fit on screen, or Ctrl+wheel when they overflow. Shift+drag or Shift+wheel pans. Double-click resets. The arrow keys, +, - and 0 work when the plot has focus.
• The hover crosshair shows the value in effect at that time for every signal, and marks held values.
• The table below the plot gives the statistics of the visible range: samples, min, max, mean, standard deviation, peak-to-peak and first→last change.
• PNG saves the plot. CSV saves the visible samples of every plotted signal.
• Millions of samples stay fast: each pixel column draws the min/max of its samples. Zoom in far enough and the raw samples and their markers appear.

Spectrum

• A spectrogram of any vector column: time × frequency, in dB or linear, with the viridis, magma, blue or gray colormap.
• The peakN_hz columns are drawn on top as dots.
• Click to pick a window, or drag to average a time range. The spectrum below shows the selection over the average of all windows, and labels the three highest peaks.
• Drag on the spectrum to zoom the frequency range. The wheel zooms time, and the time range is shared with the Plot tab.

Statistics

• One row per column: samples, sample rate, held %, min, max, mean, std, median, P5, P95, RMS, NaN count, a trend sparkline of the window means, and a small histogram.
• Sort by any column and filter by name. Constant signals are flagged with CONST (and can be hidden in the signal list too).
• Select a row for its histogram with P5, median and P95 marked, the full percentiles, window counts, and its distinct values when it is a state signal such as Forge Mode.
• Held samples are left out by default, because a held value is not a new reading. Change this per view or with etlExplorer.stats.includeHeld.

Correlation

• Pearson r between the per-window values of the plotted signals, or of up to 40 varying signals. A series uses its window mean, a scalar its value, and a vector the RMS of its spectrum.
• Click a cell for a scatter plot with a regression line.

Data

• Windows: one row per window with start, date/time, duration and the window value of each plotted signal.
• Samples: every stored sample of one signal, with its held flag.
• Either table can be limited to the plot's visible range and exported as CSV.

Files

• Every file of the item with its role, owning column, dtype, shape and size.
• Select a file to see its NumPy header and first values, or the manifest text.

Settings

Each view also remembers its own choices (plotted signals, zoom, tab and so on) while the editor stays open.

Performance

Arrays are read only when a view needs them, and they are cached per item, up to 512 MB, shared by all editors on the item. The overview reads only the manifest, t0/dur and the small held and offsets files. Statistics for every column are computed when you open the Statistics tab: about 3 s for an item with 154 columns and 20 million samples.

Development

npm install
npm test           # unit tests (npy parser, store, stats) incl. the real item in ../ETL when present
npm run harness    # renders the real webview in headless Edge against ../ETL/261021-000-01.etl, writes images/
npm run test:e2e   # runs the extension inside VS Code against a synthetic item
npm run package    # builds etl-explorer-<version>.vsix (private distribution; never published)

Set ETL_ITEM=<folder> to run the harness against another item.