Skip to content
| Marketplace
Sign in
Visual Studio Code>Machine Learning>Llama AIONew to Visual Studio Code? Get it now.
Llama AIO

Llama AIO

Timo Leser

|
492 installs
| (0) | Free
All-in-one local llama.cpp for VS Code: install, HF models, LM Studio-style settings, shared server, Copilot Chat provider.
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

Llama AIO for VS Code

Run local LLMs with llama.cpp and use them in GitHub Copilot Chat — without leaving VS Code.

Llama AIO installs llama.cpp for you, finds or downloads GGUF models, tells you whether a model fits on your GPU before you load it, and runs one llama-server that all your VS Code windows share.

Llama AIO sidebar: server status, selected model, and memory fit verdict Llama AIO sidebar: speed, cache reuse, context usage, Copilot Chat and llama.cpp backend

Get started

  1. Open the Llama AIO view in the activity bar.
  2. Install llama.cpp (Vulkan, CUDA or CPU — picked for your hardware).
  3. Choose a model: Change… lists GGUF files already on disk (Llama AIO, LM Studio, Strata, Unsloth, Hugging Face cache) or searches Hugging Face. First time? Use the one-click starter model.
  4. Press Start, then pick Llama AIO: … in the Copilot Chat model picker.

Know if it fits — before you load it

The Memory & load card estimates VRAM and RAM at full context and gives one clear answer: fits, tight, or will spill to system RAM. When it doesn't fit, it offers one-click fixes such as a shorter context, using an idle second GPU, or a smaller KV cache — each checked against the estimate before it's shown.

Memory verdict: over VRAM, with one-click fixes

  • Presets: Coding agent, Max context (largest context that fits), Max quality.
  • Main controls: context length, GPU layers, multi-GPU split, CPU offload for MoE/FFN layers.
  • Advanced: threads, batch sizes, KV cache types, flash attention, reasoning, RoPE and speculative decoding (MTP, DFlash, n-gram). Filter by name or flag, or show only what you changed.

Newer llama.cpp builds also support a GPU expert cache, draft sampling, and a separate draft batch size. Llama AIO checks the active binary before using these flags. Expert caching defaults to off because it needs extra VRAM; draft batch size defaults to inheriting the main batch. Draft sampling defaults to Auto, which enables probabilistic MTP when supported and keeps DFlash's existing behavior. See performance settings for compatibility and tuning details.

Advanced settings also offer a prompt-cache RAM limit (blank inherits the server default, 0 disables it, -1 is unlimited) and independent projector and draft devices. Device options use llama.cpp's device IDs; Auto follows the main model, and a separate draft can run on CPU. Unsupported controls are inactive on older builds; remote servers manage their own load settings. New llama.cpp servers can also advertise reasoning levels through /props, so an unfamiliar model name can still expose its reasoning picker without requiring a curated sampling profile.

See what's running

The header always shows the loaded model, context, GPU layers, speed and memory. Change a setting while the server runs and it lists exactly what will change — apply with Reload, or Discard to go back.

Header listing a pending change with Reload and Discard

The Performance card shows generation and prompt speed, cache reuse, speculative acceptance, and what fills your context (tools, tool results, history…).

Copilot Chat

The running model appears in Copilot Chat as Llama AIO: …, including Agent mode with streamed replies and tool calls. Curated sampling modes are applied automatically for well-known model families; otherwise set temperature, top-p/k and max tokens under Chat Requests → Sampling.

Thinking modes also work in Copilot CLI sessions. That picker uses standard effort names: for Qwen3.8 / Flash-Next, XHigh, Medium, Low, and None select the corresponding thinking presets and No Think. Existing chats with the older preset names remain supported.

Profiles also cover Spark X2.5/X2.6 (including X2.6 1.7B), GPT-OSS, GLM-5.3/Flash, DeepSeek V4.1 Flash, Tiel-Coder, MiniCPM5, FrogNano 4B 2609, Nanbeige4.3 33B, Mellum2.1, MiMo Distill, Saluki, Flint, Sage, selected Ornith and Nemotron checkpoints, LFM2.5, LFM3 1.7B, Bonsai 2, and Qwen Coder/Instruct variants. See the profile catalog for exact versions, defaults, sources, and runtime requirements.

The llama icon is supplied for both the provider and its models. VS Code 1.142 ignores the provider icon, and its Copilot CLI bridge does not carry custom model icons into CLI sessions, so those surfaces show VS Code's generic icon. The extension cannot override those host controls.

Good to know

  • Strata: the sidebar detects Desktop's public API on 47831, the standalone server on 8095, the launcher on 8080, and the source-run host on 8791, then offers Connect. Other ports are discovered when process information is available. Served model names, canonical aliases, and Desktop model labels select the matching chat profile, including Swift 1.5 and short IDs such as q2_0 for Qwen3.8 Flash-Next. Desktop chooses a different free port if 47831 is occupied.
  • Other local servers: the sidebar also offers Connect for servers it recognises on their default ports: Ollama (11434), LM Studio (1234), vLLM (8000), SGLang (30000), TabbyAPI (5000), KoboldCpp (5001), BigMoeOnEdge (8765), Project Maya (8080, 8095) and RustingStrata (8080). A llamafile, bmoe-server or rustingstrata process is found on any port where process names are visible. Each server is recognised by a reply only that program gives, so another server on the same port is not mistaken for it; a server with an API key switched on may not be recognised. PolyStrata answers like Strata and is listed as Strata. Up to three suggestions are shown. They connect like any other OpenAI-compatible server that is not llama.cpp; to look on more ports, run Llama AIO: Look for Servers on More Ports… (also linked in the Connect form); any OpenAI-compatible server on those ports is offered too. For anything else use Connect to a Server….
  • Remote context: temporary outages and Strata reloads keep the last known model, profile, and context visible for the current connection. Chat waits for a loaded model, and a newly reported context replaces the remembered value, including when it is smaller.
  • Large prompts: compatible llama.cpp servers count the rendered text prompt before inference and reserve the remaining space for the reply. Other servers retain estimates and decide whether the request fits. The sidebar shows “Waiting for first token…” during prefill.
  • Library refresh: use the refresh button beside the library filter after copying models into a watched folder. Filters also match GGUF filenames when metadata supplies a different display name.
  • One shared server at http://127.0.0.1:8742 for all VS Code windows. It starts in an external terminal (logs visible) or in the background, and always requires a per-user API key (~/.llama-aio-vs/runtime/local-api.key, created on first launch) so loopback stays a transport detail, not an authentication boundary. The extension and TUI send the key automatically.
  • Backends: Vulkan (default, good for AMD), CUDA (NVIDIA), CPU, or a llama-server already on your PATH. Upgrades are swapped in only after a successful download; pin a release tag or install from an archive if you need to.
  • Licences: the Hugging Face picker shows each model's licence and warns before downloading anything that isn't clearly permissive.
  • Vision: attach an mmproj projector to send images (added automatically when one sits next to the model).
  • PLE tables: for a model with per-layer embeddings, use PLE table → Choose… to pick from the PLE-only GGUFs found in your model folders (fitting tables first) or browse for one, then Reload. Clear restores the built-in table; a PLE-less backbone needs an attachment. Requires llama.cpp b12233 or a compatible build with --shard / --ple. Use a table from the same checkpoint; its precision may differ from the backbone. Memory estimates include the replacement table once. See PLE settings.
  • LoRA adapters: under Model → LoRA adapters, Add… lists the LoRA adapter GGUFs found in your model folders and next to the model, best match first, and attaches the one you choose; Browse for a file… opens any other GGUF. Adding or removing a file needs Reload; strength and on/off apply to the running server at once. An adapter made for a different model family is refused, one that does not name its base model gets a warning, and llama-server makes the final check when it loads. Adapters are cleared when you switch models. Adapter files are kept out of the model list. Files in .safetensors format must be converted to GGUF with llama.cpp first. The Model card also notes adapters built into a model file, and the base model a file names for itself; a LoRA that was merged into a model's weights cannot be detected.
  • Files: everything lives under ~/.llama-aio-vs/ — binaries, models, config.json, and the server log.

Settings

Run Llama AIO: Open Configuration File to edit ~/.llama-aio-vs/config.json. Useful keys in the app section:

Key What it does
port / host Server address (default 8742 / 127.0.0.1)
modelsDir / extraModelDirs Where models are downloaded / extra folders to scan
extraServerPorts More ports on this machine to look on for a server to connect to, e.g. [8001, 9000]
hfToken Token for gated or private Hugging Face models. VS Code stores the token in Secret Storage (sidebar → Hugging Face token) and migrates an existing value out of config.json; the TUI reads HF_TOKEN / HUGGING_FACE_HUB_TOKEN or this key.
backend auto, vulkan, cuda, cpu, rocm, …
launchMode externalTerminal (default) or background
autoStart Start the server when VS Code opens

All commands are in the Command Palette under Llama AIO: — start/stop/reload, open log, copy the server command line, install llama.cpp, download models, and view the last Copilot request or response.

Requirements

  • VS Code 1.109 or newer
  • GitHub Copilot Chat for the chat integration
  • code-server, VSCodium and other editors that install from Open VSX can run the server, model library and downloads, but Copilot Chat isn't available there. The models can still be used by other extensions that support the VS Code Language Model API.
  • Linux, Windows or macOS; a GPU is optional

Building and releasing

Run npm ci, npm run compile, and npm run package:vsix to build the extension into dist/.

The Build VSIX GitHub Actions workflow checks, tests, and packages every pull request and push to main. Publishing a GitHub release with a tag matching the package version (for example, v0.3.3) also attaches the VSIX to that release. Update all workspace package versions and the lockfile together before releasing. To retry an existing release, run the workflow manually with its tag.

License

MIT © 2026 Timo Leser

  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
  • Your Privacy Choices
  • Consumer Health Privacy
© 2026 Microsoft