Benzi
|
| Resolved | 412 / 500 — 82.4% |
| Total cost, all 500 instances | $12.96 |
| Cost per instance resolved | $0.031 |
| Attempts per instance | 1 (pass@1) |
| Model turns (total / median) | 14,167 / 21.5 |
Full technical report and per-instance breakdowns: GitHub.
How it works
- Compile. Tree-sitter parses every file; imports are resolved, class ancestry built, every identifier traced to its definition. The output is an index, not a blob of text.
- Query. The agent answers questions and plans edits through structured tools over that index —
profile,get_callers,backflow,trace_path,skim_source, and ~30 more. - Edit, gated. Every write passes syntax and semantic gates against the real language parser — a broken parse auto-reverts, checking blast radius before a change lands, not just after.
- Verify by running it. A focused repro runs under a runtime tracer — real argument values, real returns, real dispatch — which both proves the change and settles the map.
- Reindex, incrementally. The map follows the filesystem, never the agent's account of what it did — each turn re-parses only the files whose bytes actually moved.
Tools
A sample of Benzi's 35+ tools — the full list is on GitHub:
| Tool | What it answers |
|---|---|
get_callers |
Every call site that reaches a function — the code that will feel a change. |
call_tree |
The transitive call closure from one function, forward or in reverse. |
trace_path |
The call chain connecting two functions, and the data carried along it. |
backflow / forwardflow |
Where a wrong value came from / where a return value ends up. |
profile |
The full 360 on one symbol in a single call. |
search_symbols |
Case-insensitive substring search across every symbol in the repo. |
get_hierarchy |
A type's resolved bases and its direct subclasses. |
skim_source |
A body's one-level outline, so you know which lines are worth reading. |
execute_from / execute_generated_testcase |
Runs code under the call tracer; writes and runs a self-contained repro. |
Full list of 35+: GitHub.
Features
- Six states, never a guess — every call site and file carries one: resolved, external, candidate, unresolved, observed, or unindexed. Whatever static analysis can't settle is flagged as unsettled, not guessed.
- Runtime tracer — hooks every call during execution and overlays the observations back onto the static map.
- Persistent memory — durable per-repo facts survive restarts.
- Dual-engine: code + markup — a separate index for HTML/CSS/DOM-JS.
- Model-agnostic — Anthropic, OpenAI, or any compatible API.
Language support
Python · JavaScript · TypeScript · Java · C# · C++ · C · Go · Rust · Ruby
One compiler, ten languages. Python is deepest — it's the only one with the runtime tracer. Depth varies elsewhere; see Language support on GitHub for specifics.
Install
Search Benzi in the VS Code Extensions panel, or install from the Marketplace directly. It ships its own Python, so there's nothing else to install.
After install, the Benzi settings panel opens in the sidebar (the b icon): pick a model, paste your API key, and log in with your email. Then click Run Benzi. You can come back to that panel any time to switch models, update a key or change accounts.
This writes one shared config file — if you also use the headless CLI or MCP later, you won't need to log in again. Edits are reversible: every change Benzi makes is confined to your workspace and recorded, so it can be rolled back.
Benzi is also available as a live web demo (no install, no key needed) and, via pip install benzi, as a headless CLI and over MCP.
FAQ
Do I need an API key? For the extension, the CLI, and MCP, yes — bring your own Anthropic or OpenAI-compatible key. In VS Code you paste it into the Benzi settings panel in the sidebar; it's saved only on your machine. Only the browser demo runs without one.
Where do I change my model, key or account later? Click the b icon in the sidebar to open the Benzi settings panel. More commands, like stopping or restarting the server (in case it hangs): open the Command Palette (Ctrl+Shift+P) and type "Benzi...".
How is this different from Cursor, Copilot, or Claude Code? They find code by searching text — grep, or embedding similarity. Benzi resolves it first: a tree-sitter compiler builds a real index of symbols, call edges, inheritance and data flow, and the agent queries that index instead of guessing which files to read.
Full FAQ (privacy, pricing, CodeGraph comparison, repo size limits): GitHub.
Full README on GitHub — benchmarks, demos, and more.