Throttle Proxy
Throttling for custom endpoint models in VS Code Chat.
When you bring your own model to VS Code Chat through a custom, OpenAI-compatible
endpoint, and that endpoint limits how many requests you may send, it answers
HTTP 429 ("too many requests") once the limit is reached. VS Code Chat does not
wait and retry: it shows the error, and an agent's multi-step answer stops
halfway. This extension runs a small HTTP server on 127.0.0.1 that VS Code
sends its model requests to, and that:
- forwards requests one at a time, in order of arrival, so the gateway never
sees parallel calls from this workstation;
- on HTTP 429 waits and retries by itself: first after the initial delay,
then twice as long each time, never longer than the maximum delay. A
Retry-After (or retry-after-ms) header from the gateway is honoured, up to
that same maximum. The 429 arrives before any answer, so a retry is invisible
to VS Code; after the last retry the 429 is passed on;
- passes everything else through unchanged, streamed answers included;
- drops a request whose client has gone away (the chat's stop button) instead of
retrying it;
- answers HTTP 502 when the gateway is unreachable.
No runtime dependencies, and no API key is stored: the chat requests carry their
own, which the proxy passes through and never writes to disk.
Install
From the Marketplace (Extensions view, search "Throttle Proxy"), or from a
.vsix file:
code --install-extension throttle-proxy-0.1.0.vsix
or in VS Code: Extensions view, … menu, Install from VSIX….
Use
Open the Throttle Proxy icon in the Activity Bar (the left bar, with
Explorer and Search). The view has:
| Part |
What it does |
| Switch |
Turns the proxy on or off, at once. Remembered: with it on, the proxy starts with VS Code. |
| Status |
Running, Off, Not configured (no upstream yet), Standby (see below) or Error; and how many requests are queued. |
| Restart |
Stops the proxy and starts it with the current settings. Requests in flight are dropped. |
| Settings |
Upstream URL (no default; the proxy will not start without it), local port, initial delay, max retries, max delay, timeout. Changes take effect when the proxy restarts: while it runs, a change shows "You changed the configuration. Restart the proxy to apply it." in the view, a notification with a Restart button, and "restart needed" in the status bar. A proxy that is not running yet (no upstream, port taken, error) simply tries again with the new settings. Below them, and applied at once: what to log (All requests or Retries only, see Log) and Keep request details. |
| Log |
What it holds is set under Settings. All requests: every proxied request, every retry and the proxy's own events. Retries only: retries, the outcome of each request that needed them ("succeeded after 3 retries, 4.5 s waiting in total"), failures and the proxy's own events. Open opens the log file, Clear empties it and deletes the request details. Each request shows ✅ (answered with 2xx/3xx), ❌ (an error status, a 429 given up on, or no answer at all) or ↩️ (a 429 that was retried); a cancelled one has none. Chat requests (a JSON body with a messages array, whatever the path) are marked CC, embedding requests (a path ending in /embeddings or /embed) EMB. An entry whose request details are saved has an icon on the right; click it to open what the request sent and got back (see below). |
The status bar shows the same state; click it to open the view.
Then point your custom endpoint models at the proxy: set the upstream URL to the
endpoint's address, and in each model's URL replace that address with
http://127.0.0.1:8765, keeping the path. For example
https://llm-gateway.example.com/v1 becomes http://127.0.0.1:8765/v1. Models
that still use the endpoint's own URL bypass the proxy and are not throttled.
All settings are ordinary user settings under throttleProxy.* (Settings,
search "Throttle Proxy"), so they survive restarts and can be set in
settings.json too:
"throttleProxy.enabled": true,
"throttleProxy.upstream": "https://llm-gateway.example.com",
"throttleProxy.initialDelayMs": 1500,
"throttleProxy.maxRetries": 20,
"throttleProxy.logLevel": "retries"
| Setting |
Default |
|
enabled |
false |
run the proxy |
upstream |
(none) |
endpoint base URL, http or https |
port |
8765 |
local port, on 127.0.0.1 only |
initialDelayMs |
300 |
delay before the first retry; doubled for each further retry |
maxRetries |
5 |
retries before the 429 is passed on |
maxDelayMs |
30000 |
cap on any single delay |
timeoutSeconds |
600 |
how long the gateway may send nothing before the request fails |
logLevel |
all |
all or retries; applies at once |
keepDetails |
true |
keep each request's headers and bodies so a log entry can show them; applies at once |
With 20 retries starting at 1.5 s, a throttled request may wait about eight
minutes in total before the proxy gives up. VS Code may stop waiting first.
The log
One JSON record per line, throttle-proxy.log.jsonl in the extension's global
storage folder (Open shows the exact file). Only the newest 300 records are
kept, trimmed as each one is written; every hour, records older than one day are
deleted too. Records hold time, method, path, status and timings, never headers
or bodies.
Request details
With Keep request details on (the default), the proxy also saves what each
request sent and got back, one file per request under …/details/ next to the
log. The icon at the right of a log entry opens it as a rendered Markdown page:
- the outcome, timings, retries and the upstream URL;
- the request headers, and for a chat request the model, settings, tool names
and every message in plain text, then the raw body;
- every attempt: each 429 with its headers and body, then the final response;
for a streamed chat-completion answer, the answer put back together (content,
reasoning, tool calls with their arguments,
finish_reason, usage), then the
raw stream.
Any other API gets the same page with headers and raw bodies (JSON
pretty-printed), without the chat sections. Binary or compressed bodies (images,
audio, gzip) are not printed, only their type and size.
Headers whose names suggest credentials (authorization, *api-key*,
*token*, *secret*, cookie, password) are saved as (removed), so the
API key never reaches disk. Bodies are saved up to 1 MB each. The
conversation does reach disk, so turn the option off where that is not
acceptable. Details older than a day are deleted, and only the newest 300
requests are kept. A retry entry's icon opens a page that ends at its own 429:
the request, the attempts before it and that 429, nothing that came later. The
icon appears once its request has finished and its details are saved; the
request's final entry shows the whole history.
Several VS Code windows
Every window runs the extension, but only one can listen on the port. The first
window serves all of them, which also keeps one queue for the whole workstation.
The others show Standby, show the same log and try the port every 10
seconds, so one of them takes over when the serving window closes. Standby also
appears when another program holds the port. A
settings change is applied by restarting in the window that serves the proxy;
that is where the restart message appears.
Develop
npm install
npm test # compile and run the tests; no network needed
npm run package # throttle-proxy-0.1.0.vsix
The tests run the proxy against a fake gateway on localhost (429 sequences,
Retry-After, serialisation, client hang-up, unreachable upstream, what is
captured and what is removed), log and detail pruning, and the detail page. A
live test against a real OpenAI-compatible endpoint runs when its settings are in
the environment; the key never goes into the repository:
THROTTLE_LIVE_UPSTREAM=https://llm-gateway.example.com \
THROTTLE_LIVE_API_KEY=... \
THROTTLE_LIVE_MODEL=your-model-id \
node --test out/test/live.test.js
src/proxy.ts, src/logStore.ts and src/details.ts do not import vscode, so
they run and are tested under plain Node; src/extension.ts wires them to settings, the status
bar and the view in media/.
License
MIT, Copyright (c) 2026 Alexander Smirnov. The full text is in the LICENSE file.