Skip to content
| Marketplace
Sign in
Visual Studio Code>AI>Throttle ProxyNew to Visual Studio Code? Get it now.
Throttle Proxy

Throttle Proxy

Alexander Smirnov

|
1 install
| (0) | Free
Throttling for custom endpoint models in VS Code Chat: a local proxy that queues requests and retries HTTP 429 with backoff.
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

Throttle Proxy

Throttling for custom endpoint models in VS Code Chat.

When you bring your own model to VS Code Chat through a custom, OpenAI-compatible endpoint, and that endpoint limits how many requests you may send, it answers HTTP 429 ("too many requests") once the limit is reached. VS Code Chat does not wait and retry: it shows the error, and an agent's multi-step answer stops halfway. This extension runs a small HTTP server on 127.0.0.1 that VS Code sends its model requests to, and that:

  • forwards requests one at a time, in order of arrival, so the gateway never sees parallel calls from this workstation;
  • on HTTP 429 waits and retries by itself: first after the initial delay, then twice as long each time, never longer than the maximum delay. A Retry-After (or retry-after-ms) header from the gateway is honoured, up to that same maximum. The 429 arrives before any answer, so a retry is invisible to VS Code; after the last retry the 429 is passed on;
  • passes everything else through unchanged, streamed answers included;
  • drops a request whose client has gone away (the chat's stop button) instead of retrying it;
  • answers HTTP 502 when the gateway is unreachable.

No runtime dependencies, and no API key is stored: the chat requests carry their own, which the proxy passes through and never writes to disk.

Install

From the Marketplace (Extensions view, search "Throttle Proxy"), or from a .vsix file:

code --install-extension throttle-proxy-0.1.0.vsix

or in VS Code: Extensions view, … menu, Install from VSIX….

Use

Open the Throttle Proxy icon in the Activity Bar (the left bar, with Explorer and Search). The view has:

Part What it does
Switch Turns the proxy on or off, at once. Remembered: with it on, the proxy starts with VS Code.
Status Running, Off, Not configured (no upstream yet), Standby (see below) or Error; and how many requests are queued.
Restart Stops the proxy and starts it with the current settings. Requests in flight are dropped.
Settings Upstream URL (no default; the proxy will not start without it), local port, initial delay, max retries, max delay, timeout. Changes take effect when the proxy restarts: while it runs, a change shows "You changed the configuration. Restart the proxy to apply it." in the view, a notification with a Restart button, and "restart needed" in the status bar. A proxy that is not running yet (no upstream, port taken, error) simply tries again with the new settings. Below them, and applied at once: what to log (All requests or Retries only, see Log) and Keep request details.
Log What it holds is set under Settings. All requests: every proxied request, every retry and the proxy's own events. Retries only: retries, the outcome of each request that needed them ("succeeded after 3 retries, 4.5 s waiting in total"), failures and the proxy's own events. Open opens the log file, Clear empties it and deletes the request details. Each request shows ✅ (answered with 2xx/3xx), ❌ (an error status, a 429 given up on, or no answer at all) or ↩️ (a 429 that was retried); a cancelled one has none. Chat requests (a JSON body with a messages array, whatever the path) are marked CC, embedding requests (a path ending in /embeddings or /embed) EMB. An entry whose request details are saved has an icon on the right; click it to open what the request sent and got back (see below).

The status bar shows the same state; click it to open the view.

Then point your custom endpoint models at the proxy: set the upstream URL to the endpoint's address, and in each model's URL replace that address with http://127.0.0.1:8765, keeping the path. For example https://llm-gateway.example.com/v1 becomes http://127.0.0.1:8765/v1. Models that still use the endpoint's own URL bypass the proxy and are not throttled.

All settings are ordinary user settings under throttleProxy.* (Settings, search "Throttle Proxy"), so they survive restarts and can be set in settings.json too:

"throttleProxy.enabled": true,
"throttleProxy.upstream": "https://llm-gateway.example.com",
"throttleProxy.initialDelayMs": 1500,
"throttleProxy.maxRetries": 20,
"throttleProxy.logLevel": "retries"
Setting Default
enabled false run the proxy
upstream (none) endpoint base URL, http or https
port 8765 local port, on 127.0.0.1 only
initialDelayMs 300 delay before the first retry; doubled for each further retry
maxRetries 5 retries before the 429 is passed on
maxDelayMs 30000 cap on any single delay
timeoutSeconds 600 how long the gateway may send nothing before the request fails
logLevel all all or retries; applies at once
keepDetails true keep each request's headers and bodies so a log entry can show them; applies at once

With 20 retries starting at 1.5 s, a throttled request may wait about eight minutes in total before the proxy gives up. VS Code may stop waiting first.

The log

One JSON record per line, throttle-proxy.log.jsonl in the extension's global storage folder (Open shows the exact file). Only the newest 300 records are kept, trimmed as each one is written; every hour, records older than one day are deleted too. Records hold time, method, path, status and timings, never headers or bodies.

Request details

With Keep request details on (the default), the proxy also saves what each request sent and got back, one file per request under …/details/ next to the log. The icon at the right of a log entry opens it as a rendered Markdown page:

  • the outcome, timings, retries and the upstream URL;
  • the request headers, and for a chat request the model, settings, tool names and every message in plain text, then the raw body;
  • every attempt: each 429 with its headers and body, then the final response; for a streamed chat-completion answer, the answer put back together (content, reasoning, tool calls with their arguments, finish_reason, usage), then the raw stream.

Any other API gets the same page with headers and raw bodies (JSON pretty-printed), without the chat sections. Binary or compressed bodies (images, audio, gzip) are not printed, only their type and size.

Headers whose names suggest credentials (authorization, *api-key*, *token*, *secret*, cookie, password) are saved as (removed), so the API key never reaches disk. Bodies are saved up to 1 MB each. The conversation does reach disk, so turn the option off where that is not acceptable. Details older than a day are deleted, and only the newest 300 requests are kept. A retry entry's icon opens a page that ends at its own 429: the request, the attempts before it and that 429, nothing that came later. The icon appears once its request has finished and its details are saved; the request's final entry shows the whole history.

Several VS Code windows

Every window runs the extension, but only one can listen on the port. The first window serves all of them, which also keeps one queue for the whole workstation. The others show Standby, show the same log and try the port every 10 seconds, so one of them takes over when the serving window closes. Standby also appears when another program holds the port. A settings change is applied by restarting in the window that serves the proxy; that is where the restart message appears.

Develop

npm install
npm test              # compile and run the tests; no network needed
npm run package       # throttle-proxy-0.1.0.vsix

The tests run the proxy against a fake gateway on localhost (429 sequences, Retry-After, serialisation, client hang-up, unreachable upstream, what is captured and what is removed), log and detail pruning, and the detail page. A live test against a real OpenAI-compatible endpoint runs when its settings are in the environment; the key never goes into the repository:

THROTTLE_LIVE_UPSTREAM=https://llm-gateway.example.com \
THROTTLE_LIVE_API_KEY=... \
THROTTLE_LIVE_MODEL=your-model-id \
node --test out/test/live.test.js

src/proxy.ts, src/logStore.ts and src/details.ts do not import vscode, so they run and are tested under plain Node; src/extension.ts wires them to settings, the status bar and the view in media/.

License

MIT, Copyright (c) 2026 Alexander Smirnov. The full text is in the LICENSE file.

  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
  • Your Privacy Choices
  • Consumer Health Privacy
© 2026 Microsoft