Panda — Voice Companion

A living 3D AI companion inside VS Code.
She talks, reacts, remembers, and has real personality.

"Because coding alone at 2am shouldn't feel lonely."

What is Panda?

Panda puts a fully animated 3D AI companion into your VS Code sidebar. Hold a button, speak — she transcribes, thinks, and talks back. Her face reacts to her own emotions in real time. She remembers things you tell her across sessions. She has her own distinct personality, and she owns it.

This is not a chatbot widget. It is a complete voice pipeline with a living VRM avatar — expressions, lip sync, blink, and idle motion all driven in real time through the full conversation stack.

The Pipeline

Hold button / Space  →  mic opens
        |
   SoX binary  →  16kHz mono WAV  (node-record-lpcm16)
        |
   Whisper STT  →  transcript  (Groq · whisper-large-v3-turbo)
        |
   Compressed memory injected into system prompt
        |
   LLM  →  streamed reply + [emotion:X] tag  (Groq · llama-3.3-70b-versatile)
        |
   Emotion tag  →  avatar expression driven live
        |
   Memory compressed in background  →  key:value tokens persisted
        |
   Orpheus TTS  →  WAV audio buffer  (Groq · canopylabs/orpheus-v1-english)
        |
   Web Audio API  →  decoded, played back in webview

Every stage is wired end-to-end with no human-in-the-loop. Speak, hear her respond, watch her face change — the entire loop happens in under a few seconds.

Features

Screenshots

Default · Dark theme	Default · Light theme
Custom wallpaper · Room background

Personalities

Each personality is a completely different system prompt, not just a tone modifier. Switching characters changes how she thinks, speaks, and reacts.

Custom characters you upload get their own personality assigned through the creation wizard — completely separate from Yuriko's settings.

Setup

1 — Install SoX

Panda records audio through the sox binary. Install it before anything else.

# macOS
brew install sox

# Ubuntu / Debian
sudo apt install sox

# Windows
# Download from: https://sox.sourceforge.net

2 — Get a Groq API Key

Panda runs on Groq's inference API — it's fast and has a generous free tier.

1. Go to console.groq.com and create an account
2. Generate an API key under API Keys
3. Keep it — you'll paste it into Panda on first launch

3 — Accept Orpheus TTS Terms

Orpheus is the voice model. You need to accept the terms once before it works.

Open this link while signed into your Groq account:
console.groq.com/playground?model=canopylabs/orpheus-v1-english

Click Accept Terms when prompted. You only need to do this once.

4 — Install the Extension

Install from the VS Code Marketplace, or press F5 in this repo to launch the Extension Development Host.

5 — First Launch

1. Click the Panda icon in the VS Code activity bar
2. Paste your Groq API key when prompted
3. Choose your companion from the onboarding screen
4. Hold the mic button (or press Space) and start talking

Adding Your Own Character

Panda supports custom VRM models through a built-in creation wizard.

1. Open Settings → Character → Switch Character
2. Click Add Your Own
3. Select any .vrm file from your computer
4. Name your companion and choose their personality
5. Hit Create — Panda loads them with the full animation set

Your custom character retains their own name, personality, and profile. Switching back and forth between characters loads each one's settings independently.

Settings Reference

All settings persist across sessions. Memory is compressed by the LLM into short key:value tokens rather than storing raw chat logs — efficient and private.

Memory System

Panda remembers things you tell her, but not by storing chat logs. After each conversation turn, a background call to a small, fast LLM (llama-3.1-8b-instant) compresses the exchange into a compact token string:

name:alex|work:coding|music:rap|wake:730|mood:tired

This string is re-injected into every system prompt. Yuriko uses it naturally — she doesn't recite it back, she just knows.

Memory persists to disk across VS Code restarts. You can view the current summary or clear it entirely from the Memory settings tab.

Tech Stack

Development

# Install dependencies
npm install

# Compile TypeScript
npm run compile

# Bundle VRM scene
npm run bundle

# Watch mode
npm run watch

Press F5 in VS Code to launch the Extension Development Host with live reloading.

The extension requires the sox binary at runtime for audio capture. All other dependencies are npm packages.

Architecture

Extension Host (Node.js)              Webview (HTML/JS)
  src/panel.ts         ←——————————→   media/main.js
  src/groqClient.ts                   media/style.css
  src/audioCapture.ts                 media/vrm-bundle.js
  src/memoryManager.ts                webview/index.html
  src/secretManager.ts

All audio I/O lives in the Extension Host. Web Speech API and getUserMedia are not available in VS Code webviews — Panda routes everything through the Node.js layer and communicates with the webview via postMessage.

Roadmap

• [ ] Multiple concurrent custom character slots
• [ ] More facial expression triggers and idle behaviors
• [ ] Localization support
• [ ] Additional built-in companions (Animal Kingdom Vol. 2+)