SkillFoundry VS Code Plugin

Behavioral telemetry agent for technical candidate assessments

Overview

The SkillFoundry VS Code Plugin is a privacy-first telemetry extension that captures behavioral signals during technical assessments. Unlike traditional coding tests that only measure test pass/fail, we analyze how developers work to generate 105 behavioral signals across 10 dimensions.

Key Features

BYO-IDE: Candidates work in their own VS Code with their own tools and workflow
In-IDE Communication: Chat with AI Customer and TPM personas directly within the plugin
SIEM-grade Telemetry: 105 signals derived from real development behavior
Privacy-First: Metadata only, no code content until final submission
No Cloud Code Runner: Candidates run code locally, eliminating infrastructure complexity

Quick Start

For Candidates

Receive Assessment Link: Your hiring team will send you a unique assessment link
Install Extension: Click the link or search for "SkillFoundry" in VS Code marketplace
Enter Session Token: Paste the token from your assessment link
Clone Repository: The plugin will prompt you to clone the task repository automatically
Start Working: Code as you normally would—the plugin captures behavioral signals
Chat with Stakeholders: Ask questions to the AI Customer and TPM in the chat panel
Submit: Click "Submit Assessment" when complete (or time expires)

For Developers

See DEVELOPMENT.md for setup instructions.

Architecture

┌──────────────────────────────────────────────────────┐
│                 Candidate's Machine                  │
│                                                      │
│  ┌────────────────────────────────┐  ┌────────────┐  │
│  │          VS Code              │  │ Cloned Repo │  │
│  │                               │  │ (GitHub)    │  │
│  │  ┌─────────────────────────┐  │  │             │  │
│  │  │   SkillFoundry Plugin   │  │  │ Task files  │  │
│  │  │                         │  │  │ Tests       │  │
│  │  │  ┌───────┐ ┌─────────┐ │  │  │ README      │  │
│  │  │  │Collect│ │ AI Chat │ │  │  └────────────┘  │
│  │  │  │ ors   │ │ Panel   │ │  │                   │
│  │  │  └───┬───┘ └────┬────┘ │  │                   │
│  │  └──────┼──────────┼──────┘  │                   │
│  └─────────┼──────────┼─────────┘                   │
│            │          │                              │
│            │ Events   │ Messages (REST)              │
└────────────┼──────────┼──────────────────────────────┘
             │          │
             ▼          ▼
┌──────────────────────────────────────────────────────┐
│              SkillFoundry Backend                    │
│                                                      │
│  ┌──────────┐  ┌───────────┐  ┌───────────────────┐  │
│  │ Event    │  │ Session   │  │ AI Conversation   │  │
│  │ Ingest   │  │ Manager   │  │ Engine            │  │
│  │ (REST)   │  │           │  │ (Customer & TPM)  │  │
│  └────┬─────┘  └─────┬─────┘  └────────┬──────────┘  │
│       │              │                  │             │
│  ┌────▼─────┐  ┌─────▼──────┐  ┌───────▼──────────┐  │
│  │ Event    │  │ Signal     │  │ Conversation     │  │
│  │ Store    │  │ Processor  │  │ Store            │  │
│  │ (TimeSe) │  │ (105 sigs) │  │                  │  │
│  └────┬─────┘  └─────┬──────┘  └───────┬──────────┘  │
│       │              │                  │             │
│       └──────────────┼──────────────────┘             │
│                      ▼                                │
│              ┌───────────────┐                        │
│              │ Report Gen    │                        │
│              │ (AI Engine)   │                        │
│              └───────────────┘                        │
└──────────────────────────────────────────────────────┘

See ARCHITECTURE.md for detailed design.

105 Behavioral Signals

The plugin captures events that feed 105 signals across 10 dimensions:

Dimension	Signal Count	Examples
Task Outcome & Code Quality	10	Test pass rate, coverage delta, complexity
Problem Solving & Strategy	10	Exploration patterns, refactoring discipline
Execution Discipline	15	Test frequency, linting, incremental commits
Collaboration & Communication	10	Clarifying questions, tone, response time
Requirements & Product Thinking	10	Early spec reading, edge case discovery
Risk Management & Safety	10	Security patterns, error handling
Velocity & Time Management	10	Time to first action, work pattern consistency
Environment & Tool Mastery	10	IDE shortcuts, git sophistication, debugger use
Learning Behavior	10	Doc usage, search quality, self-checking
Professionalism & Reliability	10	Commit cleanliness, communication gaps

See SIGNAL-CATALOG.md for the complete list.

AI Customer & TPM Chat

Candidates can chat with two AI personas during the assessment:

AI Customer

Role: End-user or business stakeholder who requested the feature
Personality: Non-technical, focused on outcomes and UX
Behaviors: Answers questions about requirements, clarifies edge cases, introduces ambiguity to test requirement-gathering

AI TPM (Technical Program Manager)

Role: Project coordinator who assigned the task
Personality: Semi-technical, focused on timelines and scope
Behaviors: Provides context about priorities and constraints, pushes back on scope expansion, acknowledges status updates

Chat interactions generate signals for Collaboration, Requirements, and Professionalism dimensions.

Privacy Model

What We Collect ✅

Event metadata (timestamps, types, counts, durations)
Edit shapes (line ranges, character deltas, not content)
Terminal commands and output patterns
Git metadata (commit messages, file counts)
Chat messages with AI personas
Final git diff at submission (for code analysis)

What We Do NOT Collect ❌

Source code content during editing
Keystroke logging
Screen recording or screenshots
Clipboard content
AI prompt/completion content
Personal files outside assessment workspace
Network traffic monitoring
Other application monitoring

See PRIVACY.md for complete privacy policy.

Project Structure

SkillFoundry-VSCode-Plugin/
├── src/
│   ├── extension.ts              # Extension activation
│   ├── session/                  # Session management
│   ├── collectors/               # Telemetry collectors
│   ├── chat/                     # AI Customer/TPM chat
│   ├── transport/                # Backend communication
│   ├── ui/                       # Status bar, task view
│   ├── privacy/                  # Sanitizer, scope guard
│   └── analysis/                 # Test framework detection
├── webview/                      # Chat panel UI (HTML/CSS/JS)
├── test/                         # Unit and integration tests
├── docs/                         # Documentation
└── package.json                  # Extension manifest

Development Phases

Phase 1: Core MVP (Current) ✅

Extension scaffold, session management, timer
Core collectors: File, Edit, Terminal
REST transport with batch event submission
Basic chat panel UI
Backend: Event ingestion, AI chat

Phase 2: Full Telemetry (Next)

Remaining collectors: Git, Debug, Search, Focus, Clipboard, AIAssistant
Test framework detection (jest, pytest, mocha, go test)
Full 105-signal coverage

Phase 3: Analysis & Reporting

Post-submission diff analysis pipeline
Signal scoring engine
Report generation with AI narrative

Phase 4: Polish & Scale

Hiring team dashboard
Assessment template library
ATS integration

Phase 5: IntelliJ (Future)

Port to IntelliJ Plugin SDK

Documentation

ARCHITECTURE.md - Detailed system design
BACKEND-INTEGRATION.md - API specs and integration guide
SIGNAL-CATALOG.md - Complete list of 105 signals
PRIVACY.md - Privacy policy and data transparency
TESTING-STRATEGY.md - Unit/E2E test approach
DEVELOPMENT.md - Developer setup guide
CONTRIBUTING.md - Contribution guidelines

License

This is proprietary software. Unauthorized copying, distribution, or use is strictly prohibited.

Contact

Issues: Use GitHub Issues for bug reports and feature requests
Questions: Reach out via internal Slack (#dev-ide-plugin)
Security: security@skillfoundry.dev

Built with ❤️ by the SkillFoundry team

SkillFoundry Assessment

SkillFoundry