Dataform Toolkit for VS Code

SQLX language support, a lineage graph, compile + BigQuery dry-run cost on save, run-from-sidebar and config linting for Dataform. Works with the local CLI or the Google Cloud Dataform API.

Features

Includes in the graph

Dataform's compile output doesn't record which includes/*.js files a definition uses, so the extension scans source text: require("includes/x") (and relative requires) plus global usage like constants.LIMIT / dates.fn() (each includes file is a global named after the file). Include nodes feed the actions that use them, and other includes that require them. It's a heuristic: comments are ignored, but a local variable with the same name as an include can cause a false positive. Includes are found on compile, so save the file (or run Compile Project) after adding one.

Backends

Switch with Dataform: Switch Backend (or the status-bar item).

• CLI – uses dataform compile --json / dataform run. Needs @dataform/cli (or set dataform.cli.command to npx --yes @dataform/cli) and, for real runs, .df-credentials.json (dataform init-creds bigquery). If the project has a package.json but no node_modules, the extension runs dataform install first.
• Cloud – uses the Dataform API with Application Default Credentials (gcloud auth application-default login). Set dataform.cloud.project, .location, .repository. By default (dataform.cloud.gitCommitish: "auto") it compiles the branch you have checked out, and recompiles when you switch branches. Set a branch / tag / SHA to pin it, or set dataform.cloud.workspace to compile a Cloud workspace instead. Runs create workflow invocations and stream per-action states to the Dataform output channel.

Cloud compiles what is on the remote, not your working tree. So on a local branch:

• Not pushed → refused up front with the exact command (git push -u origin <branch>); nothing is sent to the API.
• Local commits not pushed / uncommitted edits → compiles the pushed tip and says so (status-bar tooltip, warning colour) so you don't mistake it for your local state.
• Detached HEAD → uses the commit if a remote branch contains it.
• A warning if your checkout's origin differs from the Cloud Dataform repository's connected remote.

The pushed/unpushed check uses your last git fetch (it never touches the network). Compile-on-save and dry-run-on-save remain CLI-only; for instant feedback on local edits use the CLI backend.

Sidebar structure, includes and external tables

Group Actions By… builds a multi-level tree (dataform.tree.hierarchy, outermost first) from: layer, project, schema (dataset), folder, type, tag. Presets cover the common cases, e.g. Layer › Project › Dataset › Type for one-project-per-layer setups, or Layer › Folders.

• folder expands the definition file path into real nested sub-folders (definitions/bronze/extract/… → bronze › extract). dataform.tree.folderRoots (default ["definitions"]) sets which leading folders are stripped.
• Layers are ordered upstream → downstream and shown with their colour; assertions sit with the table they check.
• Declarations are shown by default (dataform.tree.hideTypes is empty), so external tables in your bronze project are part of the structure. Hide types with e.g. ["assertion"].
• Includes are items in the tree. Each lists the actions that use it and the includes it requires, and opens its file on click. dataform.includes.folders (default ["includes"]) chooses which folders are scanned, e.g. ["includes", "external_tables"]. dataform.tree.includes: root (an Includes folder), byLayer (an Includes folder inside each layer that uses them, transitively; needs layer as the first level), or off. The same includes appear as nodes in the lineage graph, so you can see how an external-table definition feeds bronze actions. Only files under includes/ are Dataform globals (matched by name.fn() usage); other folders are matched through require().

Choosing what you are looking at, and lineage diff

Dataform: Choose What to View… switches the sidebar and graph between your working tree / checked-out branch (default; includes uncommitted edits with the CLI backend), Live (what your schedules run: a release configuration's compilation), or any branch, tag or commit. Non-working views are read-only snapshots: the status bar says so, and Run is disabled so you can never run one ref's code while looking at another.

Compare With… (or the Compare menu in the lineage toolbar) diffs the view against the default branch (dataform.diff.baseRef, default auto = origin/HEAD, else main/master), Live, or any ref. The lineage graph then shows:

Toggle Changed + impacted only to hide everything else, or use Show Only Changed + Impacted in the sidebar. Sidebar items get the same markers, and a Removed vs … group. Open diff on a changed action opens a file diff (base on the left, your working file on the right).

How it works: the other ref is compiled in a temporary git worktree (your checkout is never touched; cached per commit) with the same Dataform CLI, or through the Cloud API when the Cloud backend is active. Actions are matched by schema.name, so a dev-project and prod-project compile still line up. SQL is compared whitespace-insensitively.

Exporting graphs

The lineage toolbar has an Export… menu (and, when the view is filtered, a scope switch: What's shown / Whole graph).

Text formats can be saved or copied. Limits: GitHub's Mermaid renderer refuses graphs over ~500 edges (the export says so in a comment: export a filtered view); a PNG of a very large graph is scaled down to fit browser canvas limits, and refused with advice if it would be unreadable (SVG always works). Everything the webview hands back is validated before it touches disk (SVG must be script-free, PNG must be a real PNG, size-capped), and files are only written where you choose in the Save dialog.

Layer colours

With layers configured, the graph colours nodes by layer as well as by action type: the type stays as the left stripe and legend, the layer adds a coloured band, a tint and a label, with its own Layer key (click a layer in the key to hide / show it). Defaults: bronze copper, silver grey, gold yellow, source slate blue; anything else gets a stable palette colour. Override per layer with "color": "#ff8800". The toolbar's Colour menu chooses type + layer / layer / type. Layer groups in the sidebar carry the same colour dot.

Layers (bronze / silver / gold) and team configuration

Layers are not built in; you define them. A layer is a name plus matchers. An action belongs to the first layer whose defined matchers all match:

Declarations that match no layer are source; assertions take the layer of the table they check. Layers drive the lint rules (layer-dependency, requireAssertions, …), the layer label on graph nodes, and Group Actions By → layer (ordered upstream → downstream).

Run Dataform: Set Up Layers to build this from your compiled project: pick project / schema / folder, tick the values that are layers, name them, and give the order. Use the same name for several values (e.g. dev + prod projects) to put them in one layer.

// dataform-toolkit.json — layers per GCP project, dev + prod in the same layer
{
  "tree": { "groupBy": "layer" },
  "lint": { "layers": [
    { "name": "bronze", "project": "(^|-)bronze(-|$)", "allowedDependencies": ["source", "bronze"] },
    { "name": "silver", "project": "(^|-)silver(-|$)", "allowedDependencies": ["bronze", "silver"], "requireAssertions": true },
    { "name": "gold",   "project": "(^|-)gold(-|$)",   "allowedDependencies": ["silver", "gold"], "requireAssertions": true, "requireDescription": true }
  ] }
}

Repo-level configuration (dataform-toolkit.json)

Commit dataform-toolkit.json next to workflow_settings.yaml and every clone and every branch gets the same setup with no manual steps. Dataform: Set Up Repo for the Team creates it (pre-filled from workflow_settings.yaml) and adds the extension to .vscode/extensions.json so VS Code suggests it to anyone who opens the repo. The file gets autocomplete and validation in VS Code.

• Precedence: your own VS Code settings (user / workspace) → dataform-toolkit.json → extension defaults. The file supplies team defaults; personal choices still win.
• Safe by construction: only a whitelist of settings can come from the file (backend, cloud.project|location|repository|workspace|gitCommitish, compile.vars, dryRun.project|location|pricePerTiB|warnAboveGiB, lint.*, tree.*, run.*). Anything that runs commands (cli.command), points at credentials, moves the project root, or enables schedule editing is ignored and logged: a cloned repo is untrusted input.
• Workspace Trust: in an untrusted workspace the file is not applied and compiling/running is disabled (compiling executes the repo's JavaScript).
• Branches: the file lives in git, so it follows the branch (and dry-run costs, graph, lint and includes are recomputed per branch; costs are kept only while an action's SQL is unchanged). The extension also watches definitions/, includes/, workflow_settings.yaml and the config file, so switching branches, pulling or rebasing recompiles automatically (CLI backend).
• Codespaces / dev containers: see examples/sample-project/.devcontainer/devcontainer.json for a starting point (installs the Dataform CLI; you still authenticate to Google Cloud yourself). .vscode/extensions.json and dev-container extension lists resolve extensions by publisher.name, so set a real publisher in package.json and install from the Marketplace or a private gallery; a bare .vsix has to be installed once per machine.

Schedules & Logs (Cloud Dataform) are always LIVE

The panel is unmistakably marked: a red LIVE badge, a red frame, a banner naming the repository, and a louder EDITING IS ENABLED banner when writes are on. Every confirmation says LIVE. Each schedule shows code from: <git ref> (its release configuration's ref, with when it last released, and a warning if that release is disabled or failing), because scheduled runs use that code, not whatever branch you have checked out. Choosing tags/actions for a schedule reads the live release's compilation, never your current branch (with an announced fallback if it can't be read).

Open with Dataform: Show Schedules / Show Run Logs (or the calendar / history icons in the sidebar title). Works from the dataform.cloud.* settings regardless of which backend is active; auto-refreshes while visible (dataform.logs.autoRefreshSeconds).

What changes Cloud Dataform, and how it's guarded

Edits are sent as field-masked updates (only the field you changed), and existing settings such as the service account are preserved. If your schedules are managed in Terraform/IaC, editing them here will drift from that source.

GoogleSQL support

Dataform only runs GoogleSQL (BigQuery SQL), so the extension is strict about it rather than assuming T-SQL/ANSI.

• Highlighting: a GoogleSQL grammar inside .sqlx and .bqsql files. Functions that exist in other databases but not in BigQuery are shown as invalid immediately.
• Checks as you type (no compile or credentials needed; on by default, switch off with dataform.lint.googleSql):

Severity per rule (or family, e.g. sql-syntax) is set in dataform.lint.rules. Most problems have a quick fix (lightbulb / Ctrl+.); Fix all applies every unambiguous one in the file.

• Suppress a false positive: -- dataform-lint-disable-next-line sql-unknown-function, or -- dataform-lint-disable <rule> ... -- dataform-lint-enable (# and // comments work too).
• Custom rules: dataform.lint.customRules (also settable in dataform-toolkit.json):

"dataform.lint.customRules": [
  { "id": "no-current-timestamp", "pattern": "CURRENT_TIMESTAMP\\(\\)", "severity": "error", "layers": ["gold"],
    "message": "Use ${vars.run_ts} so reruns are reproducible", "replace": "${vars.run_ts}" },
  { "id": "gold-needs-docs", "kind": "config-key", "keys": ["description", "tags"], "layers": ["gold"], "message": "gold actions need $key" }
]

The built-in function list is maintained by hand, so a brand-new BigQuery function can be flagged as unknown (an information hint by default): add it to dataform.lint.extraFunctions.

Cost of a change

With a comparison active (Compare With...), Estimate cost of change (graph toolbar, or the command palette) runs BigQuery dry-runs for the new and changed actions and everything downstream of them. It reports a normal run (incremental tables use their incremental query) and a full refresh, with and without downstream actions; views build for free. Actions that cannot be estimated (typically an upstream table that does not exist yet in the target dataset) are listed and the totals are marked as a lower bound. The result also appears in the Markdown change report. Dry runs are free but need read access to the tables involved.

Large projects

Above 400 actions the lineage graph opens as an overview with one node per layer (or use Group: by layer + dataset), edges aggregated and diff counts rolled up. Click a group to expand it in place; Collapse all resets. Only what is on screen is drawn, and far away nodes become plain boxes. Without grouping, drawing stops at 4,000 actions.

Lint rules

Configure under dataform.lint.*. Each rule can be error | warning | information | hint | off.

GoogleSQL syntax rules and custom rules are described in GoogleSQL support.

Layers default to bronze / silver / gold (matched on schema name) with source for declarations – edit dataform.lint.layers to match your project:

"dataform.lint.layers": [
  { "name": "extract", "schema": "^extract", "allowedDependencies": ["source"] },
  { "name": "bronze",  "schema": "^bronze",  "allowedDependencies": ["source", "extract", "bronze"] },
  { "name": "silver",  "schema": "^silver",  "allowedDependencies": ["bronze", "silver"], "requireAssertions": true, "namePrefix": "" },
  { "name": "gold",    "schema": "^gold",    "allowedDependencies": ["silver", "gold"], "requireAssertions": true, "requireDescription": true }
]

Cost estimate caveats

• Uses BigQuery dryRun and dataform.dryRun.pricePerTiB (default 6.25 USD/TiB on-demand). It is bytes processed, not your bill – capacity/editions pricing, clustering pruning and cached results differ.
• Dry-running a model whose upstream tables don't exist yet fails with Not found; build upstream first.
• Only tables, views, incrementals and assertions are dry-run (not operations/DDL).
• Estimate cost of change sends one dry-run per action (two for incrementals), de-duplicating identical SQL and caching results for 10 minutes.

Trying it

See TESTING.md for a staged test plan and examples/sample-project for a small project that exercises every feature.

Development

npm install
npm run typecheck && npm test    # pure-logic unit tests (parsing + lint)
npm run build                    # bundles to dist/extension.js
npm run package                  # produces dataform-toolkit-<version>.vsix

Press F5 in VS Code (with this folder open) to launch an Extension Development Host.

Install the packaged file with code --install-extension dataform-toolkit-0.1.0.vsix.

Roadmap ideas

Column-level lineage, per-action last-run status from Cloud invocations, a select-star quick-fix from dry-run schemas, a naming quick-fix, autocomplete for GoogleSQL functions and ref().