Toy Models Gate

Local HTTP gateway that routes AI coding assistants inside VS Code (Claude Code, Codex, GitHub Copilot, Cline, Continue, Devin, etc.) to custom or self-hosted generative-AI models — local runtimes (Ollama, LM Studio, llama.cpp) and private/cloud deployments (OpenAI, Anthropic, Azure OpenAI, and any OpenAI-compatible endpoint).

The gateway listens on 127.0.0.1 and speaks the wire protocols clients already know — Anthropic Messages, OpenAI Chat Completions, OpenAI Responses, and Ollama — then forwards traffic to your configured upstreams.

Features

• Role profiles as models — an alias can carry a system_instruction, so qwen-react and qwen-backend appear as separate entries in your client's model picker.
• Effort normalization — translate client-side effort vocabulary (high, minimal, …) into whatever your upstream deployment actually accepts (xhigh, medium, low), with fallback / reject / passthrough policies and a VS Code warning on fallback.
• Per-client tokens & usage accounting — one token per client; every request is recorded to ~/.toymodelsgate/usage/YYYY-MM-DD.jsonl with client, workspace, alias, tokens, latency and errors.
• Stateless — no conversation state; the client owns history, provider prompt-cache hints (cache_control, prompt_cache_key) pass through untouched.
• Per-workspace config — <workspace>/.toymodelsgate/config.yaml overrides ~/.toymodelsgate/config.yaml.

Commands

All commands are available under toymg.* (and the toymodelsgate.* aliases):

Configuration

Quick setup

The fastest path is the guided Settings Panel — no YAML required:

1. Ctrl+Shift+P → Toy Models Gate: Open Settings Panel.
2. API Keys tab → Register a secret: give it a name (e.g. my-super-model-key) and paste the key value. Values are stored in VS Code SecretStorage and never displayed again.
3. Providers tab → Add provider: name, type, base_url (include /v1 for OpenAI-compatible endpoints) and the API key name you just registered. Hit Check connection to verify reachability before saving.
4. Models tab → Add model: pick an id (what clients will request), the provider and the upstream model name. Optionally expand Role profile for a system instruction and Effort policy for effort mapping; choose the Default model for unresolved requests.
5. Gateway tab → only if you need a different port or auth: none.
6. Clients tab → Set Up Clients writes the Claude Code / Codex configuration for you; each client gets its own token.
7. Save configuration in the footer — on first save you'll be asked where to store it (workspace / VS Code profile / machine). Reload refreshes the panel from disk at any time.

Manual setup

Prefer editing by hand? The same configuration lives in YAML — either use the Advanced YAML tab inside the Settings Panel, or open the file directly (Toy Models Gate: Open Configuration File). After changing the file manually, run Rediscover Models (or Reload in the panel) so the running gateway picks it up.

~/.toymodelsgate/config.yaml — three scopes merge, most specific wins: <workspace>/.toymodelsgate/config.yaml → VS Code profile → user home. Providers merge by key, models by id. The Settings Panel edits the most specific existing file, or asks where to save on first use:

providers:
  my-super-model:
    type:
      openai-compatible # openai | anthropic | azure-openai | ollama
      # lm-studio | llama.cpp | openai-compatible
    base_url: https://example.internal/v1
    key: secret:my-super-model-key # stored in VS Code SecretStorage
    # timeout_ms: 120000 # optional — upstream idle/TTFT timeout (ms), default 120s
    retry:
      retries: 3
      incrementalDelayInMs: 500
      retryWhenStatusIs: [429, 502, 503, 504]
      doNotRetryWhenStatusIs: [400, 401, 403, 404, 422]
      # connectionRetryErrorCodes: socket codes retried by default
      # (ETIMEDOUT, ECONNRESET, ECONNREFUSED) — the request never reached
      # the server, so status rules don't apply. [] disables.

effort:
  default_on_invalid: fallback # fallback | reject | passthrough

models:
  - id: qwen
    use: my-super-model/qwen-N.M
    effort:
      supported: [xhigh, medium, low]
      default: xhigh
      on_invalid: fallback
      map:
        high: xhigh # client 'high' → upstream 'xhigh'

  - id: qwen-react
    use: my-super-model/qwen-N.M
    system_instruction: >
      You are a senior frontend engineer specialized in React, TypeScript
      and Node.js.

default: qwen

server:
  host: 127.0.0.1
  port: 4000
  auth: required # 'none' for clients that cannot send a token

All effort keys are optional — when omitted, the gateway passes the client's effort hint through untouched and the upstream is responsible for validation.

Example: adding a role model

Say you want a qwen-golang role — same upstream model, a Go-focused system instruction and its own effort policy:

providers:
  my-super-model:
    type: openai-compatible
    base_url: https://example.internal/v1
    key: secret:my-super-model-key

models:
  # …existing models (qwen, qwen-react, …) stay listed above…
  - id: qwen-golang
    use: my-super-model/qwen-N.M
    system_instruction: >
      You are a senior Go engineer. Prioritize idiomatic Go, explicit error
      handling with wrapped errors (fmt.Errorf with %w), context propagation,
      table-driven tests, and minimal dependencies. Prefer the standard
      library over third-party packages unless strictly necessary. Follow
      the project's existing module layout and respect go.mod conventions.
    effort:
      supported: [xhigh, medium, low]
      default: medium
      on_invalid: fallback
      map:
        high: xhigh
        minimal: low

default: qwen-golang

Register the key once (Toy Models Gate: Register API Key or the Settings Panel → API Keys tab), then clients can request model: "qwen-golang". Each alias carries its own role profile — duplicate one per stack.

Client setup

Run Toy Models Gate: Set Up Clients to auto-write:

• Claude Code — ~/.claude/settings.json: env (ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN, CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY, ANTHROPIC_SMALL_FAST_MODEL), availableModels + enforceAvailableModels over your aliases, model set to the default alias, and per-alias modelSettings.effortLevel from each effort.default. With one alias it also registers ANTHROPIC_CUSTOM_MODEL_OPTION/_NAME (keeps the client effort selector). With two or more it writes a modelPicker.replaceBuiltInOptions lineup — one labeled row per alias — and ANTHROPIC_DEFAULT_{OPUS,SONNET,HAIKU,FABLE}_MODEL pins so internal model slots (background tasks, subagents, opusplan) resolve to your default alias. Requires Claude Code ≥ 2.1.242 for modelPicker.
• Codex — ~/.codex/config.toml (model_providers.tmg block; set TMG_API_KEY env var to the client token)

For Copilot and other clients, point the client at http://127.0.0.1:4000 (Ollama-compatible endpoints /api/chat, /api/tags are emulated) with the token shown via Register Client Token.

Compatibility

Any client that accepts a custom base_url and a bearer token can use the gateway. Officially supported:

Client aliases appear as models in each client's picker, each with its own system_instruction and effort policy. Per-client setup details, config examples and a capability matrix: COMPATIBILITY-EN.md · COMPATIBILITY-ES.md.

Troubleshooting

401 — "Missing or invalid gateway token"

Every request is checked by strict comparison against the per-client tokens stored in VS Code SecretStorage (token:claude-code, token:codex, …). The 401 response body and the Output channel log include a fingerprint of the token the client sent (first 12 chars + length) — if it shows 'mg-claude-…' instead of 'tmg-claude-…', you already see the typo.

On activation the extension also checks ~/.claude/settings.json for token/port drift and offers a one-click Re-sync. Common causes of a stale token:

• Token rotated via Register Client Token after the client file was written — re-run Set Up Clients (or Toggle Claude Code Gateway off/on) so the file is rewritten with the current token.
• SecretStorage doesn't persist — on Linux/WSL it relies on the OS keyring (gnome-keyring over dbus). If auth breaks again on every restart, either configure a keyring or set server.auth: none in config.yaml (the gateway only binds to 127.0.0.1).
• Hand-edited token — ANTHROPIC_AUTH_TOKEN must match the tmg-<client>-<32 chars> format exactly.

Known limitations

• AWS Bedrock and Google Vertex provider types are recognized in the schema but deferred — requests return a structured "not supported in this release" error.
• count_tokens is forwarded only to Anthropic providers.
• No server-side session memory (by design — see ADR-001 §AD-8).

Development

See CONTRIBUTING.md for the development guide — setup, commands, fake providers, architecture rules and conventions.

License

BSD 3-Clause — © Vickodev