TraceTest — Smart Test Generator

Tests based on how developers actually use their code.

TraceTest is a VS Code extension that generates realistic, high-value tests by combining:

SOURCE CODE + REAL RUNTIME EXECUTION + ACTUAL INPUTS/OUTPUTS + AI  →  REALISTIC TESTS

Table of Contents

• Why it exists
• Key differentiator
• Installation
• Quick start
• Recording workflow
• Supported runtimes
• Supported test frameworks
• AI setup
• Privacy
• Security
• Architecture
• Known limitations
• Development
• Testing
• Packaging
• Roadmap
• Contributing

Why it exists

Most AI test generators look only at source code and produce generic, speculative tests — plausible-looking assertions nobody asked for. TraceTest's core differentiator is that it generates tests from real observed usage: you run or debug your app normally, TraceTest watches what actually happens (which functions get called, with what inputs, producing what outputs, and what breaks), groups that into meaningful flows, and only then asks an AI to turn evidence-backed flows into tests. Nothing is generated "because it's theoretically possible" — every test carries a reason and a confidence score you can inspect before it touches your codebase.

Key differentiator

"Your code tells us what could be tested. Your real usage tells us what should be tested."

Installation

git clone https://github.com/AnandShah10/tracetest
cd tracetest
npm install
npm run compile

Then press F5 in VS Code (or run the "Run TraceTest Extension" launch config) to open an Extension Development Host with TraceTest loaded, or package it with npm run package and install the resulting .vsix.

Quick start

1. Open a TypeScript/JavaScript/Node.js project in the Extension Development Host.
2. Run Smart Tests: Start Recording (recording is off by default — you'll be prompted to enable smartTests.recording.enabled).
3. Run or debug your app so the code you care about actually executes.
   • JS/TS: for full call-tracing coverage, launch through the provided debug config pattern (see Recording workflow) so the Node instrumentation loader is injected.
   • Python/Django: same idea — prepend $SMARTTESTS_PY_SITECUSTOMIZE_DIR to PYTHONPATH when launching (python manage.py runserver, python manage.py test, or pytest) for full tracing plus, for pytest, real per-test pass/fail via the injected plugin.
   • Alternatively, set breakpoints and use VS Code's normal debugger (Node or debugpy) — the Debug Observer captures state at each stop.
4. Run Smart Tests: Stop Recording.
5. Open the Smart Test Generator view in the Activity Bar to see recorded flows with confidence scores and evidence.
6. Right-click (or select) a flow → Smart Tests: Generate Tests.
7. Review the preview panel: test code, confidence, evidence, observed inputs/outputs, risks, dependencies, and a diff of the target file.
8. Accept & Apply to write the test into your workspace, or Run Test to execute it first.

Recording workflow

UI controls while recording: once you start a session, a dedicated Stop Recording button (red, $(debug-stop)) appears in the status bar next to the main "Smart Tests" item — it's only visible while a session is active, so it's always obvious whether you're currently recording. The same state also drives the Tree View's title bar (Start/Stop swap automatically) and the Command Palette (only the relevant command is offered at any time).

TraceTest combines multiple observers, each with real, documented capabilities and real limitations — nothing here is simulated:

We do not claim to intercept every function call in your program in every circumstance. What we do claim is accurate: real entry/exit/exception tracing for exported JS/TS functions and all Python functions in launched-and-instrumented processes, and real DAP variable inspection at stop events for any language VS Code can debug.

Per-test granularity for pytest (fixes the JS-side limitation)

The JS TestObserver only sees whole-process pass/fail via VS Code Tasks — no stable API exposes per-test results from third-party JS test explorers. For Python, we don't have that constraint: TraceTest also injects a small pytest plugin (smarttests_pytest_plugin.py, alongside sitecustomize.py on the same PYTHONPATH) that hooks pytest's real, documented pytest_runtest_logreport hook and streams a test-result event per test node ID, with real pass/fail and failure detail — not just a whole-run exit code. Enable it by adding -p smarttests_pytest_plugin to PYTEST_ADDOPTS (TraceTest does this automatically when you launch pytest through the provided debug/task config).

Enabling full instrumentation coverage

When a recording session starts, TraceTest writes loader files and exposes their paths via environment variables so you (or your own launch.json) can inject them:

• Node/TS: SMARTTESTS_LOADER_PATH — inject via node -r "$SMARTTESTS_LOADER_PATH".
• Python: SMARTTESTS_PY_SITECUSTOMIZE_DIR — inject by prepending it to PYTHONPATH.

Example .vscode/launch.json entries in your own project:

{
  "type": "node",
  "request": "launch",
  "name": "Run app (traced)",
  "program": "${workspaceFolder}/src/index.ts",
  "runtimeArgs": ["-r", "${env:SMARTTESTS_LOADER_PATH}"]
},
{
  "type": "debugpy",
  "request": "launch",
  "name": "Run app (traced, Python)",
  "program": "${workspaceFolder}/manage.py",
  "args": ["runserver"],
  "env": { "PYTHONPATH": "${env:SMARTTESTS_PY_SITECUSTOMIZE_DIR}" }
}

or from a terminal:

node -r "$SMARTTESTS_LOADER_PATH" dist/index.js
PYTHONPATH="$SMARTTESTS_PY_SITECUSTOMIZE_DIR" python3 manage.py runserver
PYTHONPATH="$SMARTTESTS_PY_SITECUSTOMIZE_DIR" PYTEST_ADDOPTS="-p smarttests_pytest_plugin" python3 -m pytest

If you don't do this, TraceTest still records via the Debug Observer (at breakpoints, including Python via debugpy) and Static Fallback (signatures only) — just with less coverage. This tradeoff is intentional and disclosed, not hidden.

Note on the include/exclude glob matching: both the Node and Python loaders match file paths against your smartTests.recording.includeGlobs / excludeGlobs using the exact same compiled regexes (via minimatch's makeRe(), computed once on the extension side and embedded verbatim into the generated loader scripts) rather than two separate hand-rolled glob implementations — this was tightened after an earlier internal version's duplicate glob logic mismatched on root-level files and patterns crossing directory boundaries. See src/observers/globCompile.ts and src/test/GlobCompile.test.ts.

Supported runtimes

• Node.js (TypeScript and JavaScript)
• Python (including Django)
• Java — test-writing only (JUnitAdapter); no runtime instrumentation observer exists for Java in this MVP (see limitations below). The DebugObserver and StaticFallbackObserver still apply, since both are language-agnostic.

Architecture supports adding Go, C#, etc. later via the same ExecutionObserver / TestFrameworkAdapter interfaces.

Supported test frameworks

• Vitest (first-class)
• Jest (first-class)
• Mocha (first-class) — detected via .mocharc.* / a mocha key or dependency in package.json; runs via npx mocha. Verified against a real installed mocha fixture (both a passing and a failing test).
• Pytest (first-class) — real detection via pytest.ini / pyproject.toml [tool.pytest.ini_options] / setup.cfg [tool:pytest] / tox.ini [pytest] / existing test_*.py files; writes to a same-directory test_<name>.py or mirrors into a root-level tests/ package if one already exists; runs via python -m pytest.
• Django (first-class) — real detection via manage.py containing DJANGO_SETTINGS_MODULE (not just any file named manage.py); finds the owning Django app by walking up to a sibling models.py/apps.py (the actual markers Django uses); writes to the app's tests.py or an existing tests/ package following Django's real discovery rules; runs via python manage.py test <dotted.path>.
• JUnit (implemented, runTest() execution unverified — see below) — real detection via pom.xml/build.gradle/build.gradle.kts; mirrors Maven/Gradle's real src/main/java → src/test/java layout convention; generates JUnit 5 (org.junit.jupiter.api) test classes. The generated .java output was compiled with a real javac and executed via reflection in this development environment, confirming the generated test structure is genuinely correct. What was not verified: the mvn test / gradle test shell-out in runTest(), since this sandbox has neither build tool nor network access to Maven Central. Treat that one method with more caution until it's been run against a real project.

AI setup

TraceTest supports four AI providers via the AIProvider interface (src/types/index.ts, registered in src/ai/AIProvider.ts) — every provider goes through the exact same privacy gating, redaction, and output validation pipeline (src/ai/generationPipeline.ts), so switching providers never trades away safety guarantees.

Choose a provider: run Smart Tests: Settings → Choose AI Provider, or set smartTests.ai.provider directly:

All API keys are stored via VS Code's SecretStorage (OS keychain-backed), never written to disk in plaintext or included in any synced settings file. The local/custom provider works without a key at all if your server doesn't require one.

Azure OpenAI specifics — real, documented differences from OpenAI's own API that this provider handles correctly rather than assuming OpenAI's shape works unchanged:

• The URL addresses your deployment, not a model name: {endpoint}/openai/deployments/{deployment}/chat/completions?api-version={version}.
• Auth uses an api-key header with the raw key, not Authorization: Bearer.
• No model field is sent in the request body — the deployment already pins the underlying model.
• If your API version or deployed model doesn't support response_format: json_object, the request is retried once without it automatically (the system prompt already demands JSON-only output, so this degrades gracefully rather than failing generation).
• Verification status: request construction (URL shape, headers, body) was built strictly from Azure's documented REST API contract and is covered by unit tests against a mocked fetch (src/test/AzureOpenAIProvider.test.ts), but has not been exercised against a real Azure OpenAI resource — this development environment has no Azure subscription and no network path to *.openai.azure.com. Treat it the same way as JUnitAdapter.runTest(): the shape should be correct, but hasn't been proven against the live service yet.

Regardless of provider:

1. Set smartTests.privacy.localOnly to false (it defaults to true, which disables all AI calls) and smartTests.ai.allowExecutionData to true if you want observed inputs/outputs included in requests — with it false, generation uses only source code and signatures, producing less specific but still evidence-plan-gated tests.
2. smartTests.ai.model controls the model string sent to the API — e.g. claude-sonnet-4-6 for Anthropic, gpt-4o for OpenAI, or whatever model name your local server has loaded.

Adding another provider means implementing AIProvider and registering it in extension.ts — no other module needs to change.

Privacy

Privacy is a first-class feature, not an afterthought:

• Recording is opt-in. smartTests.recording.enabled defaults to false.
• Redaction runs on every captured event, before it's ever written to disk: known secret shapes (API keys, JWTs, AWS keys, GitHub tokens, Slack tokens, private key blocks, DB connection strings with credentials, Bearer/ Basic auth headers) and any field whose name looks sensitive (password, token, apiKey, cookie, secret, etc. — configurable via smartTests.privacy.redactVariableNames).
• A second sanitization pass runs immediately before any payload is sent to the AI provider: one check for whole-token secret shapes (a captured value that IS a secret), and a separate unanchored scan (scanAndRedactSecrets) for secrets nested inside other text — e.g. a hardcoded API key embedded in real source code that gets sent as sourceContext.targetSource, which JSON-escaping would otherwise hide from a simple whole-token check. Both run independent of the earlier per-event redaction.
• smartTests.privacy.localOnly (default true) blocks all AI network calls outright.
• smartTests.ai.allowExecutionData (default false) is a second, explicit gate specifically for sending observed inputs/outputs (as opposed to just source code) to the AI provider.
• Recordings are stored in the extension's workspace-scoped storage directory (context.storageUri), not synced, with configurable maxEvents, maxSizeMb, and retentionDays.

Security

• Model output is always treated as untrusted. Every AI response is schema-validated (OutputValidator.validateSchema) before any part of it is used; malformed tests are rejected individually rather than failing the whole batch.
• Generated code is scanned for dangerous patterns (child_process, eval, process.exit, raw fs deletion, direct .env reads, raw https requires) and rejected if found — TraceTest never executes arbitrary AI-suggested commands or installs dependencies automatically.
• All file writes go through WorkspaceGuard, which resolves the target path and rejects anything outside the workspace root, anything inside node_modules, and dangerous executable extensions (.sh, .exe, .ps1, etc.).
• Nothing is written without explicit confirmation. The preview panel shows the full diff; only clicking "Accept & Apply" writes to disk.
• Duplicate detection (TestValidator) prevents the same test name from silently accumulating across regenerations.

Architecture

src/
├── extension.ts              — activation, wiring
├── commands/                 — command handlers (thin, delegate to modules below)
├── recorder/                 — ExecutionRecorder, ObserverManager, session lifecycle
├── observers/                — DebugObserver, NodeInstrumentationObserver, PythonInstrumentationObserver, TestObserver, StaticFallbackObserver
├── flow/                     — FlowAnalyzer, ScenarioDetector, FlowGrouper, ConfidenceScorer (all deterministic, no AI)
├── generation/                — ContextBuilder, TestPlanner, TestGenerator, TestValidator
├── ai/                        — AIProvider interface + registry, ClaudeProvider, OpenAIProvider (OpenAI + local/compatible), AzureOpenAIProvider, generationPipeline, PromptBuilder, AIResponseParser
├── frameworks/                — TestFrameworkAdapter interface, VitestAdapter, JestAdapter, MochaAdapter, PytestAdapter, DjangoTestAdapter, JUnitAdapter
├── privacy/                   — PrivacyManager, SecretDetector, Redactor
├── storage/                   — RecordingStore, FlowStore, SettingsManager
├── ui/                        — FlowTreeProvider, TestPreviewPanel, StatusBarManager
├── security/                  — WorkspaceGuard, OutputValidator
├── types/                      — shared type contracts
└── test/                       — unit tests

Data flow: raw runtime events → ExecutionRecorder (redaction, limits) → RecordingStore → FlowAnalyzer (deterministic grouping/scoring) → FlowStore → developer selects a flow in the Tree View → ContextBuilder (bounded context) → the configured AIProvider (Claude / OpenAI / Azure OpenAI / local) → OutputValidator → TestPreviewPanel → explicit Accept → WorkspaceGuard-checked write → optional runTest.

Known limitations

• Node instrumentation only traces top-level exported functions, and only in processes launched with the loader injected. It does not see private closures, or calls in processes it wasn't attached to at launch. (Python instrumentation does not share this limitation — sys.settrace sees every call, including methods and closures — but shares the "only launched-and-injected processes" constraint.)
• No runtime instrumentation observer exists for Java. DebugObserver (via any DAP-compliant Java debugger) and StaticFallbackObserver (regex- based public/protected method extraction) both apply, but there is no Java equivalent of the Node/Python call-tracing observers in this MVP — stated plainly rather than glossed over.
• The Debug Observer only captures state when execution is stopped (breakpoint/exception) — VS Code/DAP provide no generic call-tracing hook. This applies equally to Python via debugpy and Java via any DAP adapter.
• The JS TestObserver sees only whole-process pass/fail via VS Code Tasks, not per-test results (no stable API exposes that from third-party JS test extensions). Python does not share this limitation — the injected pytest plugin gives real per-test granularity via pytest's own pytest_runtest_logreport hook, verified end-to-end against a real pytest run (one passing, one failing test, both correctly reported).
• Flow grouping uses a time-proximity heuristic (default 2s gap), not true distributed tracing correlation IDs — most apps don't have those.
• JUnitAdapter.runTest() (the mvn test/gradle test shell-out) was not empirically verified — this development environment has neither Maven nor Gradle, and no network access to Maven Central. Everything else about JUnit support (detection, Maven/Gradle layout conventions, and the generated test code itself) was verified for real, including compiling the generated .java file with a real javac and executing it via reflection.
• AzureOpenAIProvider was not empirically verified against a live Azure OpenAI resource — this development environment has no Azure subscription and no network path to *.openai.azure.com. Request construction (deployment-based URL, api-version query param, api-key header, no model field in body, retry-without-response_format on a 400) was built strictly from Azure's documented REST API contract and is covered by unit tests against a mocked fetch, same honesty bar as the JUnit case above.
• The Python and Java static-fallback extractors are regex-based (not full AST parsers — this extension doesn't bundle parser dependencies for either language). They're used only for enrichment when no dynamic data exists and never contribute to confidence, so the coarseness is a low-stakes tradeoff, but it's disclosed rather than presented as equivalent to the TS/JS ts-morph-based analysis.
• PytestAdapter's generated import now correctly walks real Python package boundaries (__init__.py presence, bounded at the workspace root) rather than assuming the workspace root is always on sys.path — this fixes the common src/-layout case (src/mypackage/calc.py now correctly imports as mypackage.calc, not src.mypackage.calc). Verified against a real pytest.ini + pythonpath = src project: the generated import form actually passes.
• multiprocessing child processes and C-extension/built-in calls are not visible to the Python instrumentation observer — real CPython/sys.settrace constraints, not an oversight.
• This project has not been run inside a live VS Code Extension Development Host in this development environment (no display, and no network access to download the VS Code binary that @vscode/test-electron needs). Every individual mechanism (the generated Node/Python loader scripts, the pytest plugin, the JUnit-generated code, the glob-matching fix) was verified by actually running it standalone against real fixtures outside VS Code, and the extension code itself passes strict tsc, ESLint, and 97 unit tests — but the full command-driven UI flow (clicking "Start Recording," watching the Tree View populate from a live debug session, etc.) has not been observed running end-to-end inside VS Code itself. Before relying on this in a real project, run it via F5 in your own VS Code and exercise the full workflow once.
• Confidence scores are a documented weighted heuristic over observable factors, not a statistical or ML-derived estimate.
• AI test generation depends on an internet connection and a valid Anthropic API key; it is fully optional and gated behind two independent privacy settings.

Development

npm install
npm run watch     # incremental compile

Press F5 (or use the "Run TraceTest Extension" launch config) to open an Extension Development Host.

Testing

npm test          # vitest run — 97 unit tests covering redaction, flow
                   # analysis, confidence scoring, output validation,
                   # workspace guard boundaries, test planning, glob
                   # compilation, secret-scanning, status bar/UI state,
                   # every TestFrameworkAdapter's detection/path/import
                   # logic, and every AIProvider's request construction
                   # (via a mocked fetch for OpenAI/Azure OpenAI)
npm run test:watch

For extension-level integration testing (activating the extension in a real VS Code instance, exercising commands end-to-end), use the @vscode/test-electron harness with the "Extension Tests" launch config — write specs under src/test/integration/ targeting out/test. This MVP ships unit tests for all deterministic logic, verified against real external tools where possible (a real pytest/mocha/javac run, not just mocked); a full integration suite against a real Extension Development Host is the natural next addition (see Roadmap) — this development environment had no display and no network path to the VS Code binary @vscode/test-electron needs to download, so that specific gap could not be closed here.

Packaging

npm run package    # compiles, then runs vsce package -> tracetest-0.1.0.vsix

This was verified for real in development: vsce correctly resolves only the production dependency subgraph (ts-morph, minimatch, and their transitive deps — no typescript/eslint/vitest/devDependencies leak into the package), .vscodeignore excludes src/, tests, and dev config, and the resulting .vsix's extension/out/extension.js was extracted and require()'d directly with a stub vscode module — it loads cleanly and exports activate/deactivate as expected, confirming the bundled dependencies actually resolve at runtime, not just at compile time.

Debugging the extension itself

1. Open this repository in VS Code.
2. Press F5 — this runs the npm: compile pre-launch task, then opens an Extension Development Host with TraceTest active.
3. Set breakpoints in any src/*.ts file; they'll bind against the generated source maps in out/.
4. In the Dev Host window, open a separate Node/TS project to actually exercise recording end-to-end.

Publishing

vsce login AnandShah
vsce publish

Requires a Personal Access Token registered against the AnandShah publisher on the Visual Studio Marketplace.

Roadmap / next highest-value improvements

1. Full integration test suite against a real Extension Development Host (@vscode/test-electron) exercising the complete recording → flow → generate → apply → run pipeline end-to-end, for JS/TS, Python, and Java.
2. Verify JUnitAdapter.runTest() against a real Maven/Gradle project with network access to Maven Central — the one JUnit code path not yet empirically exercised.
3. A Java instrumentation observer (e.g. via the JVM Tool Interface or a javaagent bytecode-weaving approach) to close the "no full call capture for Java" gap the same way sys.settrace closed it for Python.
4. sys.monitoring-based Python tracing (3.12+) as a lower-overhead alternative to sys.settrace on newer interpreters, falling back to sys.settrace automatically on older ones.
5. Django REST Framework-aware test generation: recognize DRF views and generate APIClient-based tests instead of plain Client, when DRF is detected as a dependency.
6. Smarter flow grouping using explicit correlation IDs when available (e.g. OpenTelemetry trace context already present in the target app) instead of only the time-proximity heuristic.
7. Streaming responses for the AI providers that support it, so the preview panel can show a generated test filling in progressively instead of waiting for the full response.
8. Incremental re-recording: merge new observations into an existing flow's evidence instead of only ever starting fresh sessions.
9. Inline CodeLens on functions with existing flow evidence, offering one-click "Generate Test" without going through the Tree View.
10. Verify AzureOpenAIProvider against a real Azure OpenAI resource — the one AI provider code path not yet empirically exercised, same reasoning as item 2.

Contributing

Contributions are welcome — see CONTRIBUTING.md for ground rules (the short version: don't invent capabilities, verify what you can against something real, no pseudo-code). Please also see CODE_OF_CONDUCT.md and, for security-sensitive reports specifically, SECURITY.md rather than a public issue.

See CHANGELOG.md for release history, including several real bugs found and fixed during development via end-to-end verification rather than just code review.

License

MIT — see LICENSE.

Made with ❤️ by Anand Shah for the developer community