JSONL FastViewer

A high-performance, local-first VS Code custom editor engineered for massive JSONL and NDJSON files.
Progressive byte-range indexing • Adaptive sparse memory layouts • Virtualized 60 FPS UI • Crash-safe streaming save

English • 简体中文

Highlights • Performance • Visual Tour • Architecture • Usage • Configuration • Installation

Highlights

• ⚡ Engineered for Massive Datasets
  Files are never loaded whole into memory. Powered by a custom zero-copy ByteLineScanner, files up to 5 GB (43+ million rows) open within seconds via lazy byte-range reads while VS Code remains smooth and responsive.

• 🪶 Adaptive Indexing with 814× Memory Reduction
  Uses exact Float64 line indexing (~13 B/line) for instant access on small-to-medium files, and automatically transitions to a sparse checkpoint index (~0.0156 B/line) above the 256 MiB soft watermark. On a 5 GB dataset, resident index memory drops from 534 MB to just 0.69 MB while keeping 100-row window reads under 0.84 ms.

• 🎯 Three Synchronized Coordinated Views
  Seamlessly switch between a Virtualized Table (multi-column sorting, resizable columns, pinned row numbers), Raw JSONL (high-performance virtualized text view), and a JSON Tree Inspector (deep hierarchical object navigation). Selections, active cursors, and uncommitted edits stay synchronized across all views.

• 🔍 Interactive, Non-Blocking Streaming Queries
  Run instant field filters, comparisons, and text searches (e.g. score > 90, status:active, name contains "Alice"). Scanning operates at > 1,000,000 rows/second, returns first matches in ~3.5 ms, responds to cancellation in < 1 ms, and enforces strict event-loop yield budgets to eliminate UI freezing.

• ✏️ Type-Aware In-Place Cell & Record Editing
  Inline cell editing (F2), full-record JSON modification, row insertions, and row deletions with an undo/redo journal (Cmd/Ctrl+Z). Modified and inserted records display real-time status badges with immediate dirty-state tracking.

• 🛡️ Crash-Safe Streaming Save Architecture
  Zero risk of data corruption. Edits stream into a sibling temporary file using aggregated 1 MiB write buffers, undergo byte-for-byte verification and source file identity re-validation, emit exactly 1 fsync, and commit atomically via platform-verified rename. Cancelled or interrupted saves leave the original file byte-for-byte untouched.

• 🔒 100% Local-First & Private
  All parsing, scanning, indexing, querying, and saving execute entirely on your machine. Zero network requests, zero telemetry, zero background uploads.

Visual Tour

Virtualized Data Grid	Type-Aware Inline Editing
Smooth 60 FPS scrolling across millions of rows with auto-detected schemas and multi-column sorting.	Press F2 to edit inline with automatic type parsing, validation, and real-time dirty status badges.

Performance Benchmarks

All figures below are measured on Apple M4 (macOS arm64, Node v22.21.1, 16 GB unified RAM, seed 42) using the automated benchmark harness under benchmark/. Fixtures, synthetic generators, and gate runners are checked into the repository for deterministic reproduction.

1. Scaling & Pipeline Throughput (100 MB → 5 GB)

[!TIP] Extreme Line Density: On ultra-short line fixtures (e.g. 100 MB with millions of tiny lines), the byte scanner peaks at 16.9 Million rows/second framing speed (449 ms total build time), requiring zero string allocations and zero GC pauses.

2. Adaptive Memory Footprint: Exact vs. Sparse Index (5 GB File)

When indexing exceeds the configurable 256 MiB soft watermark, JSONL FastViewer seamlessly adopts a sparse checkpoint index on the fly without restarting or copying pages:

3. Real-World Edit Densities & Save Pipeline (500 MB, 4.31M Rows)

JSONL FastViewer utilizes a dedicated 1 MiB aggregated buffer writer (PhysicalLinePump + BufferedTemporaryFileWriter). Unmodified rows stream directly from raw source bytes without JSON parsing or string serialization overhead:

• Zero Write Call Thrashing: Constant ~1 MiB write blocks (avgWriteBlockBytes = 1,048,576 B) regardless of whether you edit 1 line or 1,000,000 lines.
• Atomic Single fsync: Commits with a single atomic fsync call, guaranteeing that interrupted saves never leave broken or truncated files.

4. Query, Filter & Schema Responsiveness

• Zero Cache Pollution: Batch query and schema scans run in non-caching mode, preventing LRU cache thrashing and maintaining hot cache lines for active UI views.
• Strict UI Responsiveness Budget: Event loop latency during massive 500 MB scans stays capped under 16 ms (P99: 11.7 ms), guaranteeing stutter-free typing and scrolling while scans execute in the background.

5. UI & Virtualized DOM Benchmarks (Headless Chromium DOM Gates)

Architecture

JSONL FastViewer follows a clean, decoupled architecture where performance-critical operations run with strict resource bounds and fail-safe transactional guarantees:

flowchart TB
  subgraph Presentation ["Webview Layer (Sandboxed Browser)"]
    VGrid["Virtualized Data Grid\n(Fixed ~47 DOM rows • 60 FPS)"]
    RawView["Virtualized Raw View\n(Line-level virtual text)"]
    Inspector["Hierarchical Inspector\n(Tree-structured JSON explorer)"]
    DOMState["Bounded View Store\n(Zero fs access • Strict budgets)"]
  end

  subgraph Protocol ["Typed Protocol Boundary"]
    IPC["JSON-RPC Typed Messages\n(Versioned • Validated DTOs • Bounded windows)"]
  end

  subgraph Host ["Extension Host (Orchestration & File I/O)"]
    Session["Session & Version Coordinator\n(Session ID + File Version Leases)"]
    Scanner["ByteLineScanner\n(Zero-string UTF-8 framing)"]
    Indexer["Adaptive LineIndex\n(Exact 13B/row ⇄ Sparse 0.015B/row)"]
    QueryEngine["Streaming Query Engine\n(Regex-free • Cancellable • Yield budgets)"]
    SaveCoordinator["Streaming Save Transaction\n(1 MiB aggregated buffer • fsync=1 • Atomic rename)"]
  end

  subgraph Storage ["Local Storage"]
    SourceFile[("Original .jsonl / .ndjson")]
    SiblingTemp[("Sibling .tmp.<nonce>")]
  end

  VGrid <--> IPC
  RawView <--> IPC
  Inspector <--> IPC
  IPC <--> Session

  Session --> Scanner
  Scanner --> Indexer
  Session --> QueryEngine
  Session --> SaveCoordinator

  SourceFile -. Byte-range reads .-> Scanner
  SaveCoordinator -. Aggregated streaming write .-> SiblingTemp
  SiblingTemp == Atomic commit on SHA validation ==> SourceFile

Core Architectural Invariants

1. Zero Full-File Buffering: The extension host never reads an entire file into memory as a string or parsed record array. All reads occur via bounded byte-range chunks.
2. Untrusted Webview Principle: The webview operates in an isolated environment with zero file system access. Messages are strictly validated against runtime DTO schemas.
3. Session & Version Cancellation: Every operation is tagged with sessionId + fileVersion. When a file is modified externally or reloaded, in-flight work and outdated caches are cleanly cancelled and evicted.
4. Crash-Safe Out-of-Place Writes: Edits are never applied in-place. If a crash, disk exhaustion, or power cut occurs during save, the original source file remains byte-for-byte intact.

Usage & Shortcuts

Opening Files

• Default Editor: Double-click any .jsonl or .ndjson file in the VS Code Explorer.
• Context Menu: Right-click a file and select JSONL: Open Viewer.
• Command Palette: Run JSONL: Open Viewer (Ctrl+Shift+P / Cmd+Shift+P).

Keyboard Shortcuts

Query Syntax Examples

• Field Equality: status:active or city:"San Francisco"
• Numeric Comparisons: score >= 90, age < 30, latency != 0
• Substring Matching: name contains "Alice", path contains "/api/v1"
• Compound Filters: Space-delimited clauses evaluate with implicit AND logic.

Configuration

Custom settings can be configured via VS Code Settings (Cmd+, or Ctrl+,):

Installation

You can install JSONL FastViewer from a locally built VSIX package:

# 1. Clone the repository
git clone https://github.com/bigwhites/JSONL-FastViewer.git
cd JSONL-FastViewer

# 2. Install dependencies & compile
npm install
npm run package

# 3. Install the generated VSIX into VS Code
code --install-extension jsonl-fastviewer-1.0.0.vsix

Try in Extension Development Host

To run and experiment without packaging:

npm run build
code --new-window \
  --user-data-dir /tmp/jfv-profile \
  --extensions-dir /tmp/jfv-extensions \
  --extensionDevelopmentPath="$PWD"

Development & Verification

# Strict TypeScript checks
npm run check

# ESLint & layer boundary validation
npm run lint

# Code formatting checks
npm run format:check

# Run full test suite (934 unit & integration tests)
npm test

# Build production bundles
npm run build

# Integration tests in VS Code Extension Host
npm run test:integration

Benchmark Suite

# Fast feedback benchmark (10 MB)
npm run benchmark:stage6:quick

# Full multi-shape benchmark suite (100 MB / 500 MB / wide / long / bad rows)
npm run benchmark:stage6

# Scale smoke benchmarks
npm run benchmark:stage6:smoke:1gb
npm run benchmark:stage6:smoke:5gb

# Save transaction safety & failpoint gate (79 failpoints)
npm run gate:stage5-save

# Virtualized DOM & frame latency gate (Headless Chromium)
npm run gate:stage5-dom

License

Released under the MIT License.