AI Toolkit for Visual Studio Code

What is AI Toolkit

AI Toolkit for Visual Studio Code is an extension to help developers and AI engineers to easily build AI apps and agents through developing and testing with generative AI models locally or in the cloud. AI Toolkit supports most genAI models on the market.

AI engineers can use AI Toolkit to discover and try popular AI models easily. The playground supports attachments, web search and thinking mode allowing for more interactive experimentation. They can run multiple prompts in batch mode and evaluate the prompts in a dataset to AI models using popular evaluators. Additionally, AI engineers can fine-tune and deploy AI models.

Feature	Description	Screenshot
Model Catalog	Browse and access AI models from various sources. Simplified discovery of GitHub, ONNX, Ollama, OpenAI, Anthropic, and Google models.
Playground	Interactive environment for testing AI models. Quick experimentation with model capabilities including multi-modal support.
Prompt Builder	Tools for creating and optimizing prompts. Iterative improvement of prompt engineering techniques.
Batch Run	Execute multiple prompts across selected models. Efficient testing at scale with various inputs.
Evaluate an AI model with a dataset	Test AI models against datasets using standard metrics. Measure performance with using built-in evaluators such as F1 score, relevance, similarity, and coherence or create your own evaluators.
Fine-tune	Customize models for specific use cases. Adapt models to specialized domains and requirements.

Getting started

We recommend starting with models hosted by GitHub.

• Follow the installation guide to set up AI Toolkit for your device.
• From the extension tree view, select CATALOG > Models to explore models available. We recommend to getting started with models hosted by GitHub.
• From the model card, select Try in Playground to start experimenting the capability of an AI Model.

Feedback and resources

Your feedback is crucial to shape the feature roadmap. Read our developer documentations for more features or reach us in GitHub to report an issue.

AI Toolkit ❤️ Developer Community.

Data and telemetry

The AI Toolkit for Visual Studio Code collects usage data and sends it to Microsoft to help improve our products and services. Read our privacy statement to learn more. This extension respects the telemetry.enableTelemetry setting which you can learn more about at disable telemetry reporting.