See what your DataFrame looked like at every step — not just the last print().
Runs your pandas or PySpark script unmodified and lets you click any transformation line to see the DataFrame's shape, schema, and sample data at that exact point — with instant diffs against the previous step.
⚡ The Problem: Data Pipelines are Black Boxes
When writing data transformation pipelines in pandas or PySpark, data engineers and analytics engineers face persistent debugging pain points:
Broken VS Code Debug Console: VS Code's standard debug console silently fails to print PySpark DataFrame evaluation calls (df.show() or df.printSchema()) due to process stream redirection bugs ([VS Code Python Issue #18239](https://github.com/microsoft/vscode-python/issues)).
Data Wrangler Variable Inspection Failures: Microsoft's Data Wrangler extension frequently throws variable evaluation errors when inspecting PySpark DataFrames from debug frames ("Could not retrieve variable... Please file an issue").
Driver-Only Debuggers: Databricks' VS Code extension and standard Python debuggers only step through driver execution lines without showing per-transformation data deltas or row-loss metrics.
Intrusive Logging Libraries: Community libraries like pandas-log or pdlog require developers to manually add decorators, wrap pipeline functions, or modify source code — creating opt-in friction and risking code churn.
🚀 How It's Different
Feature
DataFrame Lens
pandas-log / pdlog
Standard Debugger / Wrangler
Databricks VS Code
Zero Code Changes
✅ Yes
❌ No (requires decorators)
✅ Yes
✅ Yes
Inline Editor CodeLens & Gutter
✅ Yes
❌ No (terminal only)
❌ No
❌ No
Instant Row-Loss & Schema Diffs
✅ Yes
⚠️ Partial text log
❌ No
❌ No
Crash-Safe (Disk Streamed)
✅ Yes
❌ Memory lost on crash
❌ Session dies
❌ Session dies
PySpark + pandas Unified
✅ Yes
❌ pandas only
⚠️ Broken PySpark console
⚠️ Driver-only
Fully Offline & Privacy Safe
✅ Yes
✅ Yes
✅ Yes
❌ Requires Cloud
🔥 Features
Zero-Code-Change Instrumentation: Click ▶ Run with Time-Travel CodeLens at the top of any .py script. Your code runs unmodified.
Stacked Inline CodeLenses: Chained transformation calls on a single line (df.filter(...).groupBy(...).agg(...)) get stacked CodeLenses (👁 987 rows · 6 cols · −217 rows vs prev), each independently clickable.
Gutter Summaries & Hover Tooltips: See step-by-step row transformations right in the editor gutter.
Rich Step Detail Panel:
Schema delta table highlighting added, removed, and retained columns.
Side-by-side sample data preview with key-based row tracking.
Never calls .collect(), uses .limit(N).toPandas(), countApprox safety guard
PySpark (Cluster Mode)
Roadmap
Planned remote JSONL transport adapter
🏁 Quick Start
Install DataFrame Lens from the VS Code Marketplace.
Open any Python file containing pandas or PySpark operations.
Click the ▶ Run with Time-Travel CodeLens at the top of your file (or press Ctrl+Shift+P → DataFrame Lens: Run with Time-Travel).
Click through the inline CodeLenses on transformation lines to open step diffs and timeline views.
⚙️ Configuration Reference
Setting
Default
Description
timeTravelDebugger.sampleSize
20
Maximum number of sample rows captured in each snapshot.
timeTravelDebugger.maxExactCountRows
100000
Estimated rows threshold for exact PySpark count vs approximate count.
timeTravelDebugger.excludeColumns
[]
List of column names or globs to exclude from snapshots for privacy.
timeTravelDebugger.instrumentedMethods
(20 methods)
List of DataFrame method names to instrument.
timeTravelDebugger.retainRuns
10
Number of past execution runs retained on disk before automatic pruning.
timeTravelDebugger.autoShowTimeline
false
Automatically open Timeline view after run completion.
🏗 How It Works (Architecture)
The extension injects a lightweight Python bootstrap package (python/tt_bootstrap) via PYTHONPATH injection when running your script.
Monkey-Patching: Before your script runs, tt_bootstrap safely monkey-patches DataFrame transformation methods (filter, merge, join, groupBy, etc.).
Stack Frame Inspection: Each patched method inspects inspect.currentframe().f_back to capture the caller's filename, line number, and monotonic call order on that line.
Crash-Safe JSONL Tail-Watch: Snapshots are written incrementally as newline-delimited JSON (.jsonl) to $TEMP/tt-debugger/{run_id}.jsonl. The TypeScript extension host tails bytes live via fs.watch, updating CodeLenses and Webviews in real time. If your script crashes halfway through, all snapshots up to that point remain safely viewable.
🔒 Privacy & Safety
100% Offline & Local: No data ever leaves your local computer.
Safety Limits: PySpark data is sampled via .limit(20).toPandas() — never invoking .collect() on full datasets.
Column Exclusion: Configure timeTravelDebugger.excludeColumns: ["ssn", "credit_card", "*password*"] to ensure sensitive fields are never captured in snapshot files.
