Amazon CloudWatch Omni

Docs · Getting started

See exactly what your AI agent did, reproduce a failure in the Playground, and prove the fix with evaluations — without leaving your editor.

Amazon CloudWatch Omni works out of the box on local OpenTelemetry traces. No AWS account required. When your agent is deployed, sign in to your CloudWatch Omni space — your team's shared environment in AWS — to investigate the same traces your team sees in the CloudWatch Omni web UI.

The extension is the development-time companion to the CloudWatch Omni web UI. Both connect to the same space: use the extension for the local build-test-iterate loop, and the web UI for production monitoring, fleet-level views, and online evaluations. It is the one tool that runs the whole loop — traces, playground, datasets, evaluations — inside the IDE, on open OpenTelemetry data.

Investigate agent behavior

Open a trace and see every step of a run: model calls, tool calls, errors, and the input and output at each span.

Filter and inspect traces and sessions from local runs or from your space.

• Compare two runs side by side to find the difference that matters.
• Turn an interesting trace into a dataset example, or run evaluators against it directly.

Test changes in the Playground

Try a new prompt, model, or configuration against the same input before you change the application.

• Run a single prompt or your full agent flow from the IDE.
• Compare prompt, model, and configuration variants side by side.
• Save experiments and re-run them against datasets and evaluators.

Measure with datasets and evaluations

Turn real runs into a regression suite.

• Build datasets from observed traces, or import them.
• Version a dataset when you need a stable test set.
• Score runs with evaluators, and configure online evaluations for deployed agent traffic.

Keep prompts with your code

Store prompts and model configuration in your project. Edit prompt content, keep version history, and open any version in the Playground to test it.

Local and cloud modes

Switch modes as you work. The sidebar always shows the mode you are in and the data source it reads.

Quick start

1. Install the extension and open your agent project. No project yet? Run Omni: Get Started with Sample Project.
2. Open Omni from the Activity Bar.
3. Follow the guided setup. It configures instrumentation, model credentials, and your agent's start command — then start your agent and watch traces arrive.

To work with deployed agents instead, sign in to your space from the extension and open Trace Explorer or Session Explorer.

Requirements

Data and privacy

• Model provider credentials are stored in VS Code SecretStorage, which delegates to your operating system's keychain.
• In local mode, trace data stays on your machine in the project's .omni/ folder. The extension provides commands to include or exclude this data from git.
• In cloud mode, the extension reads and writes through your selected space, using the temporary credentials from your sign-in.