TokenCap

Website · Docs · Downloads · Benchmarks · GitHub

AI coding agents waste tokens re-discovering your codebase. TokenCap stops that.

Run one command. Every AI session starts with full project context — ranked, budgeted, and ready.

npm install -g tokencap
tokencap make

The numbers

Benchmarked across 3 real open-source repositories, 5 tasks each, 15 data points total. Tokenizer: js-tiktoken exact counts. Model: gpt-4o ($2.50/1M).

Naive grep-and-read:  ████████████████████████  820,101 tokens  →  $2.0503 / query
TokenCap capsule:     █                          46,337 tokens  →  $0.1158 / query

                      12.2× fewer tokens.  94% cheaper.  Same answer.

Full methodology and raw data: benchmark results
Reproduce: node benchmarks/run.js

What it does

Run tokencap make --full in any repo. This is what you see:

$ tokencap make --full

TokenCap: Full build requested.

TokenCap snapshot written:   ./.tokencap/snapshot.md
Included files:              47
Snapshot bytes:              198,432
Estimated tokens:            49,608
Profile:                     balanced

AI summary:                  .tokencap/graph/ai-graph-summary.md
Interactive graph:           .tokencap/graph/graph.html
Knowledge graph written:     .tokencap/graph/summary.md  (47 nodes, 83 edges)
Memory layer written:        .tokencap/memory/current.md
Security engine:             active (redaction engine v1)
Agent pointers written:      AGENTS.md, CLAUDE.md, .cursor/rules/tokencap.md,
                             .windsurf/rules/tokencap.md, .clinerules/tokencap.md,
                             .github/copilot-instructions.md, .kiro/steering/tokencap.md

── TokenCap Savings ─────────────────────────────────────
  Profile:  balanced
  Naive:    ████████████████████████  146,702 tok  $0.3668
  TokenCap: ████████████████           49,608 tok  $0.1240
  Saved:    97,094 tokens (66.2%)  $0.2428
  ─────────────────────────────────────────────────────────

Every major AI host (AGENTS.md, CLAUDE.md, .cursor/rules/, and 4 more) now points to .tokencap/agent/START_HERE.md. The next session starts with full project context — no re-discovery, no wasted tokens.

How it works

Your repo
    │
    ▼
tokencap make
    │  ├─ Ranks every file by git recency, importance score, and in-degree
    │  ├─ Enforces a token budget (default: 220 KB source)
    │  └─ Writes .tokencap/savings.json  ← exact before/after token counts
    ▼
`.tokencap/snapshot.md` ← the managed capsule
AGENTS.md             ← short start point for hosts that discover it automatically
.tokencap/savings.json  ← machine-readable savings data

The capsule is not a full dump. It's a graph-ranked selection of the files that matter most for the current state of the repo — so the model gets signal, not noise.

Features (v2.8.0)

1. Browser Companion Context Transfer (v2.8.0)

Bring local repository intelligence into browser-based AI chats without sending your code to a remote service:

• TokenCap Companion: Pair the Chrome companion with tokencap serve, select an indexed local project, and inject a bounded, redacted task pack into ChatGPT, Claude, or Gemini.
• Draft-Safe Injection: Existing composer text stays intact; TokenCap marks its inserted context and refuses duplicate injection. Every generated pack can also be previewed, copied, or downloaded.
• Fast, Bounded Local Bridge: tokencap serve pre-warms shared intelligence. Companion task packs stay in memory, include the project primer in their total budget, and do not create a context artifact for each browser request.
• Reviewable Capture Loop: Browser captures are validated, redacted, staged, and explicitly approved before becoming durable project memory.

2. Conversation-to-Repository Loop & Session Capture v1 (v2.7.0)

Turn finished AI coding sessions into small, redacted, reviewable handoffs — closing the conversation → repository loop without raw transcripts or telemetry:

• tokencap remember: Stage and review session summaries (--summary, --files, --decision, --next).
• Staged, Never Silent: Inbound captures land in .tokencap/memory/inbound/ with full provenance. Nothing modifies durable memory or ADRs until reviewed and approved via tokencap remember --approve <session-id> (or --reject / --note). Approved sessions archive to .tokencap/memory/sessions/.
• Graph-Grounded Quality Scoring: Captures are validated, bounded, redacted, and scored against the repository graph for path validity, groundedness, specificity, and novelty → confidence tier + evidence. Grounding evaluates references, not the truth of the summary.
• Handoff Surfaces: tokencap handoff exports the latest approved session as Markdown. Approved sessions surface in .tokencap/memory/current.md, agent-pack.md, and via MCP tools.
• Honest Host Support: Registry of 7 AI coding hosts (tokencap remember --hosts). No host is marked "auto" without a verified session-end hook. MCP process shutdown is never treated as a conversation-end signal.

3. Repository Build Health & Diagnostics (v2.7.0)

• tokencap health: Produces an explainable 0–100 repository health score evaluating intelligence freshness, git drift, symbol detection completeness, and test coverage.
• Actionable Remediation: Reports granular deduction factors and prints exact remediation commands (e.g. tokencap rebuild).
• MCP Integration: Build health is continuously surfaced in tokencap_overview and status diagnostics.

4. Evidence-Rich Risk Findings & Python Dependencies (v2.7.0)

• tokencap risk: Normalized risk findings (oversized-file, todo-density, possible-secret) with confidence tiers, structural evidence, scope, and recommended actions.
• Persistent Acknowledgment: Mark findings with --ack <id> or --dismiss <id>. Persistent state retains evidence without hiding findings from audits.
• Python pyproject.toml Parsing: Automatic dependency detection for PEP 621 standard declarations and Poetry dependencies.

5. Watcher Controls & Session ROI (v2.7.0)

• tokencap watch-config: Manage persistent file watching behavior (--enable, --disable, --debounce <ms>, --ignore <glob>).
• Session ROI Tracking: Computes observed per-session token savings and financial ROI aggregated across hosts in tokencap memory and MCP status.

6. High-Speed Loopback Bridge & Cold-Start Primer (v2.7.0)

Connect browser-based coding environments securely to local project intelligence:

• tokencap serve: Isolated loopback HTTP bridge (127.0.0.1) with pairing authentication (--token, --port, --daemon, --stop, --watch, --no-open).
• Project Primer (GET /primer): Provides a rich, human-written ~1.5k token orientation so a cold AI chat immediately understands the architecture, stack, and guidelines.
• Session Capture Endpoints: POST /capture stages captures from companions; POST /capture/approve and POST /capture/reject manage reviews over loopback.
• Warm Reuse (~3× Faster): Reuses query-independent graph, memory, and diff artifacts across queries within a short TTL, dropping repeat /generate response times from ~2.6s to ~0.8s.
• Model-Aware Dynamic Budgeting: Automatically scales task context to the target model's window (clamped between 4k and 24k tokens).
• Registry Hygiene & Zero Egress: Automatically prunes deleted directories from GET /projects. Strictly zero network egress.

7. Multi-Language Tier-1 AST Call Resolution

TokenCap parses code using pure Tree-sitter WebAssembly (zero native toolchain):

• Full Call Resolution: JavaScript, TypeScript, JSX/TSX, Python, Go, Rust, and Java.
• Go / Rust / Java Extraction: Resolves Go selector and method calls, Rust macro!/field/path calls, and Java constructor/method invocations.
• Precision Call Resolution: Resolves calls via import bindings, method receivers, and records unresolved symbols explicitly rather than guessing.

8. Static Taint & Security Flow Analysis

• tokencap analyze taint: Intra-procedural forward taint tracking from sources (req.body, query, params, cookies, process.argv, env) to sinks (db.query, exec, eval, fs.writeFile, etc.).
• Evidence-Based Risk: Clear HIGH/MEDIUM risk categorizations with sanitizer detection, without executing code.

9. Database & GraphQL Schema Graphs

• tokencap analyze schema: Parses SQL CREATE TABLE and GraphQL type schemas into a queryable relational graph, linking database models and types back to referencing application code.

10. Multi-Framework Component & SFC Graphs

• React & Next.js: AST-extracted RENDERS component trees, declared vs. passed props flow, App/Pages router maps.
• Vue & Svelte (.vue, .svelte): Single-file component (SFC) graphs with declared props and component render hierarchies.

11. Impact & Blast Radius Analysis

Know exactly what breaks before modifying or deleting a symbol:

$ tokencap impact src/auth/token.js:validateToken

  Direct callers      3 symbols · 2 files
  Transitive impact  17 symbols · 9 files
  Crosses boundary   src/api -> src/billing   ⚠
  Risk               HIGH — exported, 17 dependents

• tokencap impact --dead-code — lists uncalled, non-exported functions.
• tokencap impact --clusters — computes Louvain functional communities and cross-cluster coupling.

12. Guided Safe Refactoring

Modify code safely with deterministic, refuse-on-doubt transformations:

# Preview changes (dry-run by default)
tokencap refactor rename src/auth/token.js:validateToken checkToken
tokencap refactor rm-dead src/util/helpers.js:unusedHelper

# Apply atomically once verified
tokencap refactor rename src/auth/token.js:validateToken checkToken --write

• Safety by construction: Byte-exact outside the target edit ranges.
• Refuse-on-doubt: Aliased imports, shadowing, collisions, and computed accesses automatically block the plan.

13. Static Test Mapping & Gap Analysis

tokencap analyze tests                             # map symbols to exercising tests
tokencap analyze tests --gaps                      # list called-but-untested symbols
tokencap analyze tests --coverage coverage/lcov.info # ingest test reports (read-only)

14. Change Review & Local Ownership Signals

tokencap analyze review                    # review working-tree changes
tokencap analyze review --base main        # compare Git base ref against HEAD
tokencap analyze ownership --history       # opt-in local Git churn & author signals

Review packets include base-ref symbol evidence, caller candidates, test associations, and local Constitution/ADR context.

15. Evidence-Backed Code Simplification & Debt Ledger

tokencap analyze simplify                  # find unreachable code & duplicate artifacts
tokencap debt                              # update TODO/FIXME/HACK/DEBT ledger
tokencap compress AGENTS.md --write        # prose-only safe Markdown compression

Install & Quickstart

npm install -g tokencap

Requirements: Node.js 18+, Git.

tokencap make          # build + print savings summary
tokencap make --full   # full rebuild (graph + brain + agent + pointers)

Command Reference

7 Core Commands

TokenCap v2.8.0 consolidates repository intelligence into 7 core verbs:

[!NOTE] All legacy commands (tokencap capsule, tokencap graph, tokencap impact, tokencap refactor, tokencap debt, tokencap stats, tokencap mcp, tokencap security, tokencap health, tokencap risk, tokencap remember, tokencap handoff) remain supported as silent aliases for full backward compatibility.

Model Context Protocol (MCP)

TokenCap includes a built-in MCP server exposing 18 specialized tools over stdio:

tokencap serve --mcp --init              # auto-configure all detected AI clients
tokencap serve --mcp --init --client claude   # configure for a specific host
tokencap serve --mcp --health            # test server diagnostics and tools

Available MCP Tools

• tokencap_overview — High-level summary, tech stack, and risk clusters.
• tokencap_files — Graph-ranked, query-relevant files.
• tokencap_cluster — Deep intelligence on a specific functional cluster.
• tokencap_dependencies — Upstream/downstream dependency trees.
• tokencap_constitution — Rule constraints and invariants before coding.
• tokencap_impact — Blast radius and broken invariants for proposed changes.
• tokencap_execution — Phase-by-phase execution contract guidance.
• tokencap_delta — Incremental changes since last build.
• tokencap_search — Cross-layer search across all intelligence files.
• tokencap_verify — Cluster-specific test, build, and lint commands.
• tokencap_manage_adr — Manage Architecture Decision Records (.tokencap/constitution/adr.json).
• tokencap_get_screen — UI Map screen details and components.
• tokencap_get_diagram — Local architecture, impact, or diff SVGs.
• tokencap_simplify — Evidence-backed simplification candidates.
• tokencap_refactor_plan — Dry-run preview of guided refactoring plans.
• tokencap_test_map — Static test-to-symbol mapping and untested gaps.
• tokencap_review — Advisory change review packets.
• tokencap_ownership — Opt-in local Git history ownership signals.

Agent Host Compatibility

tokencap serve --mcp --init --client <host> records your selected host. Builds maintain that host's pointer file without overwriting custom edits:

VS Code & IDE Extensions

The VS Code extension provides commands to refresh Agent, Brain, Graph, and Memory intelligence independently, with automatic debounced capture on save.

code --install-extension .\tokencap-2.8.0.vsix

Codex and Claude Code Plugins

.\scripts\install-plugins.ps1

Security & Sovereignty

• Zero network egress — No telemetry, analytics, or remote calls. Only tokencap update touches the network. Enforced by test/local-only.test.js scanning all source files on every test run.
• Secret redaction — Every file read passes through a centralized redaction engine before becoming output. API keys, tokens, PEM blocks, high-entropy strings are replaced with [REDACTED:*] markers.
• Atomic writes — .tokencap/savings.json and all cache files use tmp-then-rename to prevent partial reads or corruptions.
• Zero native dependencies — Tree-sitter parsers are shipped as pure WebAssembly (WASM), keeping package size under 10 MB.

Further reading

License

MIT — see LICENSE.