GitHub Copilot LLM Gateway

Extend GitHub Copilot with open-source language models running on your own infrastructure.

About

GitHub Copilot LLM Gateway is a companion extension for GitHub Copilot that adds support for self-hosted open-source models. It seamlessly integrates with the Copilot chat experience, allowing you to use models like Qwen, Llama, and Mistral alongside—or instead of—the default Copilot models.

This extension connects to any OpenAI-compatible inference server, giving you complete control over your AI-assisted development environment.

Key Benefits

Compatible Inference Servers

• vLLM — High-performance inference (recommended)
• Ollama — Easy local deployment
• llama.cpp — CPU and GPU inference
• Text Generation Inference — Hugging Face's server
• LocalAI — OpenAI API drop-in replacement
• Any OpenAI Chat Completions API-compatible endpoint

Getting Started

Prerequisites

• VS Code 1.106.0 or later
• GitHub Copilot extension installed and signed in
• Inference server running with an OpenAI-compatible API

Step 1: Install the Extension

Install GitHub Copilot LLM Gateway from the VS Code Marketplace.

Step 2: Start Your Inference Server

Launch your inference server with tool calling enabled. Here's an example using vLLM:

vllm serve Qwen/Qwen3-8B \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.95 \
    --host 0.0.0.0 \
    --port 42069

Verify the server is running:

curl http://localhost:42069/v1/models

Step 3: Configure the Extension

1. Open VS Code Settings (Ctrl+, / Cmd+,)
2. Search for "Copilot LLM Gateway"
3. Set Server URL to your inference server address (e.g., http://localhost:8000)
4. Configure other settings as needed (token limits, tool calling, etc.)

Note: If the server is unreachable, you'll see an error notification with a quick link to settings:

Step 4: Select Your Model in Copilot Chat

1. Open GitHub Copilot Chat (Ctrl+Alt+I / Cmd+Alt+I)
2. Click the model selector dropdown at the bottom of the chat panel
3. Click "Manage Models..." to open the model manager

4. Select "LLM Gateway" from the provider list
5. Enable the models you want to use from your inference server

Step 5: Start Chatting

Your self-hosted models now appear alongside the default Copilot models. Select one and start coding with AI assistance!

The model integrates seamlessly with Copilot's features including:

• Agent mode for autonomous coding tasks
• Tool calling for file operations, terminal commands, and more
• Context awareness with @workspace and file references

Configuration

Configure the extension through VS Code Settings (Ctrl+, / Cmd+,) → search "Copilot LLM Gateway".

Connection Settings

Model Settings

Tool Calling Settings

These settings control how the extension handles agentic features like code editing and file operations.

Tip: If your model outputs tool descriptions as text instead of actually calling tools, try setting Agent Temperature to 0.0 and disabling Parallel Tool Calling.

Recommended Models

These models have been tested with good tool calling support:

Important: Avoid Qwen2.5-Coder models for tool calling—they have known issues with vLLM's tool parser. Use standard Qwen2.5-Instruct or Qwen3 models instead.

vLLM Setup Reference

Installation

pip install vllm

Tool Call Parsers

Each model family requires a specific parser:

VRAM Requirements

Approximate memory for BF16 (full precision) inference:

Example Server Commands

Qwen3-8B (Recommended):

vllm serve Qwen/Qwen3-8B \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.95 \
    --host 0.0.0.0 \
    --port 42069

Llama 3.1 8B:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser llama3_json \
    --max-model-len 32768 \
    --host 0.0.0.0 \
    --port 42069

Quantized Model (limited VRAM):

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.95 \
    --host 0.0.0.0 \
    --port 42069

Troubleshooting

Model not appearing in Copilot

1. Verify server is running: curl http://your-server:port/v1/models
2. Check Server URL in settings matches exactly
3. Run command "Copilot LLM Gateway: Test Server Connection" from the Command Palette

"Model returned empty response"

The model failed to generate output. Try:

1. Check tool parser — Ensure --tool-call-parser matches your model family
2. Disable tool calling — Set github.copilot.llm-gateway.enableToolCalling to false to test basic chat
3. Reduce context — Your conversation may exceed the model's limit

Tools described but not executed

The model outputs text like "Using the read_file tool..." instead of actually calling tools.

1. Use Qwen3-8B or Qwen2.5-7B-Instruct (avoid Coder variants)
2. Set Agent Temperature to 0.0
3. Disable Parallel Tool Calling
4. Ensure server has --enable-auto-tool-choice flag

Out of memory errors

• Reduce --max-model-len (try 8192 or 16384)
• Use a quantized model (AWQ, GPTQ, FP8)
• Choose a smaller model

Commands

Access from the Command Palette (Ctrl+Shift+P / Cmd+Shift+P):

Support

• Issues & Feature Requests: GitHub Issues
• Discussions: GitHub Discussions

License

MIT License — see LICENSE for details.

This extension is not affiliated with GitHub or Microsoft. GitHub Copilot is a trademark of GitHub, Inc.