Visual AI Bridge

Visual AI Bridge is a local VS Code extension for visually annotating elements in a real browser and generating structured UI context for an AI coding agent.

It captures the selected element's DOM context, CSS selector, computed styles, viewport position, visible text, and a viewport screenshot containing a numbered pin. The extension writes the result to .ai-context/ inside the current project.

No hosted backend, account, analytics service, paid API, CDN, or remote Visual AI Bridge service is required.

How it works

Visual AI Bridge uses Automatic Local Proxy Injection:

1. The extension starts one combined HTTP, proxy, runtime, and WebSocket server on 127.0.0.1.
2. You enter an HTTP localhost development URL.
3. The extension opens the same path, query, and hash through its local proxy.
4. HTML responses receive an external runtime script with escaped data attributes for the original origin and a random per-target token.
5. Normal HTTP requests and development WebSockets, including Vite HMR, continue to the target development server.
6. The browser runtime sends annotations through the reserved authenticated bridge WebSocket.
7. Send to AI creates .ai-context/ and optionally starts the configured local agent workflow.

The original page URL stored with each annotation is reconstructed from the target origin plus the browser's current proxy pathname, query, and hash. This remains correct after client-side navigation.

Five-minute quick start

1. Install visual-ai-bridge-0.1.0.vsix in desktop VS Code.
2. Open and trust the web project's local folder.
3. Start the project's HTTP development server.
4. Run Visual AI Bridge: Start.
5. Enter the localhost URL and select Open in Browser.
6. Select Annotate, click an element, and enter the requested change.
7. Select Send to AI to generate .ai-context/.
8. Use Clear Session when you want to remove the annotation batch and generated files.
9. Use Visual AI Bridge: Stop to dispose the browser runtime, sockets, timers, overlays, pins, and local server.

Automatic mode does not require editing the target project. Manual injection remains available for origin-sensitive applications or strict CSP development setups.

Capabilities

Browser annotation

• Automatic HTML injection through a dedicated same-machine proxy
• Relative URL and asset behavior preserved through the proxy
• Development WebSocket and Vite HMR forwarding
• Hover inspection overlay
• Click to annotate; Alt + click selects the closest eligible parent
• Comment editor and numbered pins
• Viewport screenshots containing the numbered pin
• Screenshot thumbnails and lightbox preview in the VS Code panel
• Pin restoration and stale-selector reporting after reload
• One active authenticated browser runtime per session

Context generation

Visual AI Bridge captures:

• Page URL and title
• User instruction
• Stable CSS selector
• Sanitized opening tag and HTML context
• Visible text
• Relevant computed CSS
• Element rectangle and viewport data
• Optional PNG viewport screenshot
• Annotation ID and sequence number

Generation is serialized per workspace. Files are written to a temporary directory and atomically installed. Existing context is backed up and restored if installation fails. Transient Windows EPERM, EBUSY, EACCES, and ENOTEMPTY rename failures are retried.

Agent modes

• Generate Only — create .ai-context/ without starting an agent
• Terminal — run a configurable command in a visible VS Code terminal
• Clipboard — copy a workspace-relative instruction and optionally open an IDE command

Supported environments

Required

• Desktop Visual Studio Code 1.90.0 or newer
• A trusted local file workspace
• An HTTP development server on localhost, 127.0.0.1, or [::1]
• A browser with WebSocket support

HTTPS targets, remote production domains, mobile devices, and other computers are intentionally excluded by the loopback-only security model.

Compatibility coverage

The automated network-level smoke suite starts real HTTP and WebSocket fixture servers and verifies:

• Astro/Vite-style HTML
• React/Vite-style HTML
• Next.js development-style HTML
• A non-Vite HTTP server
• Vite HMR WebSocket forwarding
• Manual-injection fallback
• Strict nonce-only CSP failure reporting
• Original path, query, and hash preservation
• Gzip HTML transformation
• Legacy /runtime.js compatibility

These checks validate the proxy protocol and development-server patterns. Hands-on browser testing remains recommended for applications with custom authentication, cookies, service workers, OAuth callbacks, or other origin-sensitive behavior.

Automatic mode and origin behavior

The browser-visible automatic URL uses the Visual AI Bridge proxy origin. Most applications based on relative URLs work normally, but origin-sensitive behavior can differ because browser storage and security boundaries use the proxy origin.

Potentially affected features include:

• localStorage, IndexedDB, and service-worker registrations
• OAuth redirect URI checks
• absolute-origin API allowlists
• cookies with explicit Domain or security policies
• application logic that compares window.location.origin

Use manual-injection mode when the application must keep its original browser origin.

Manual-injection fallback

The legacy route remains supported:

<script src="http://127.0.0.1:43119/runtime.js" data-visual-ai-bridge></script>

For a nonce-only CSP, add the same development nonce your application authorizes:

<script
  nonce="YOUR_DEVELOPMENT_NONCE"
  src="http://127.0.0.1:43119/runtime.js"
  data-visual-ai-bridge
></script>

The development CSP must also allow the bridge WebSocket in connect-src, for example ws://127.0.0.1:43119.

Manual fallback workflow:

1. Add the script only to the development build.
2. Enter the original target URL in the Visual AI Bridge panel and select Open in Browser once so the bridge configures that exact target origin and creates a fresh token.
3. If automatic injection displays the CSP diagnostic page, open the original target URL directly.
4. The target page loads /runtime.js from the bridge; the server supplies the token from the external script response.
5. Remove the development-only integration before production.

Visual AI Bridge does not remove or relax the target site's CSP. If the policy does not allow the injected same-origin external script, automatic mode returns a clear diagnostic page with manual-mode instructions.

Complete workflow

1. Open the source folder in desktop VS Code.
2. Start the application's development server.
3. Run Visual AI Bridge: Start or open the Visual AI Bridge Activity Bar view.
4. Enter the complete local page URL.
5. Select Open in Browser.
6. Wait for Runtime connected.
7. Select Annotate.
8. Hover over an element and click it.
9. Enter the requested change and save.
10. Repeat for other elements.
11. Select Send to AI.
12. Inspect .ai-context/ or let the configured agent mode continue.
13. Select Clear Session to remove the list, pins, generated context, temporary folder, and backup.

Sending does not clear annotations. New annotations continue after the highest current number.

Generated files

.ai-context/
├── PROMPT.md
├── context.json
├── annotation-001.html
├── annotation-001.png
├── annotation-002.html
└── annotation-002.png

• PROMPT.md contains the user instructions and technical/visual references for the coding agent.
• context.json contains the complete structured payload.
• annotation-NNN.html contains sanitized element context.
• annotation-NNN.png is optional and contains the viewport screenshot.

Atomic generation may temporarily use:

.ai-context.tmp/
.ai-context.backup/

Configuration

Terminal placeholders ${promptPath} and ${promptDir} are shell-quoted automatically for Windows and POSIX shells.

Install the VSIX

In VS Code:

1. Open Extensions.
2. Select the ... menu.
3. Choose Install from VSIX....
4. Select visual-ai-bridge-0.1.0.vsix.
5. Reload VS Code when prompted.

Command line:

code --install-extension visual-ai-bridge-0.1.0.vsix

Build and validate from source

git clone https://github.com/DominceAseberos/vscode-extension.git
cd vscode-extension
npm ci
npm run package

npm run package performs:

1. strict TypeScript checking
2. ESLint with zero warnings
3. Prettier verification
4. all unit and network-level compatibility tests
5. extension, webview, and runtime builds
6. an esbuild metadata audit that rejects unexpected external packages
7. VSIX packaging with --no-dependencies

The extension bundle includes http-proxy-3; only VS Code, Node built-ins, and optional ws native accelerators may remain external.

Current automated suite:

12 test files
65 passing tests

Security and privacy

• The combined server binds only to 127.0.0.1.
• Target URLs are restricted to HTTP loopback hosts.
• A trusted local file workspace is required.
• The webview uses a nonce-based CSP.
• Each configured proxy target receives a new random token.
• The reserved bridge WebSocket validates the token before it can replace the active socket.
• WebSocket Origin must exactly match the proxy origin; only the configured target origin is additionally accepted for manual fallback.
• Other WebSocket paths are forwarded to the development server for HMR.
• Target CSP headers are preserved; incompatible strict policies produce a visible manual-mode diagnostic.
• Runtime shutdown removes event listeners, pending animation frames, pins, overlays, the comment editor, reconnect timers, and browser sockets.
• Page HTML and screenshots are never written to extension logs.
• Form values, editable text, scripts, styles, inline event handlers, and bridge UI are removed from captured HTML.
• Screenshots are limited to 4 MB and full messages to 40 MB.
• No Visual AI Bridge cloud service receives the captured data.

Known limitations

• Automatic mode changes the browser-visible origin and may affect origin-sensitive applications.
• A nonce-only or restrictive CSP may require manual injection.
• Only one browser runtime is active per bridge session.
• Only HTTP loopback applications are supported.
• Cross-origin images, video, WebGL, canvas, iframes, and advanced native controls may not render perfectly through html2canvas.
• Automatic source-component detection is not included.
• Remote workspaces, multi-root generation, collaboration, and production-site annotation are outside the current scope.

Troubleshooting

Automatic injection was blocked

The target CSP does not authorize the injected external script. Visual AI Bridge intentionally does not weaken it. Follow the diagnostic page and the manual-injection steps above, including the required script nonce and connect-src allowance.

The panel stays on “Waiting for browser”

Check that:

• the target development server is running
• the URL uses HTTP and a loopback hostname
• the configured bridge port is free
• the opened page is the Visual AI Bridge proxy URL
• manual mode's CSP permits both the runtime script and bridge WebSocket

Health endpoint:

http://127.0.0.1:43119/health

After a target is configured, the response includes its origin:

{ "ok": true, "proxyTarget": "http://localhost:3000" }

A previous browser tab disconnects

This is expected. The newest correctly authenticated runtime becomes active. A connection with a missing token, wrong token, or wrong Origin cannot replace it.

Clear Session failed

The UI is re-enabled after generated, error, cleared, clear-error, agent-error, stopped, and unexpected-extension terminal results. If filesystem cleanup still fails, check file permissions or processes holding .ai-context/, then retry.

License

MIT