Pithos Container Viewer & Explorer
A high-performance custom editor, diagnostics workbench, and interactive vector inspection extension for Pithos (.pithos, .midb) single-file vector database containers.
Designed for AI researchers, database engineers, and vector search practitioners working with quantized high-dimensional embeddings, Matryoshka representations, and hardware-accelerated sidecars.
Features
1. Interactive Container Diagnostics
- Superblock & Magic Byte Verification: Inspects
DIOGENES v2 header offsets, schema versions, vector count ($N$), dimensionality ($D$), quantization tiers, and hardware sidecar formats (FP8 E4M3, BF16, FP16).
- Embedded Metadata & Table of Contents (TOC): Parses the trailing
PITHOSDB footer and embedded Apache Arrow columnar metadata tables (document IDs, partition indexes, topical domains, timestamps).
- Integrity Validation: Verifies 64-byte aligned memory segment offsets, CRC trailers, and file bounds.
2. High-Dimensional 3D Latent Topology (UMAP)
- Background Worker Engine: Computes real-time 3D UMAP / Spectral PCA dimensional reduction inside a Web Worker without blocking the VS Code UI thread.
- Strided Billion-Scale Sampling: Automatically samples huge datasets with strided selection for fast rendering.
- Topical Cluster Legends: Automatically maps semantic cluster labels to visual color palettes.
- Interactive Orbit & Raycasting: Rotate, zoom, and click on any point in 3D space to trigger the deep vector record inspector.
- Dynamic HUD Controls: Adjust point size (0.8px – 3.5px), toggle auto-rotation, and reset views.
3. Asymmetric k-NN Similarity Search Console
- 64-Bit SWAR Bitwise Search: Evaluates 8 bytes (64 bits) per iteration with hardware-level bitwise popcount routines.
- Asymmetric FP8 QueryLUT: Precomputes Query Look-Up Tables for FP8 E4M3 float vectors to evaluate exact Euclidean and Cosine distances in sub-millisecond latency.
- Normalized Cosine Metrics: Displays true Cosine Distance ($d_{\text{cos}} \in [0, 1]$) and percentage similarity matches with natural cluster cutoffs.
4. Vector Record Inspector Drawer
- Multi-Tier Matryoshka Heatmaps: Inspect bit-packed binary signatures across individual quantization tiers (e.g. Tier 0: 64 bits, Tier 1: 128 bits).
- Decoded Sidecar Floats: Displays decoded FP8/FP16 values alongside statistical metrics: Dimension, L2 Norm, Min, Max, and Mean.
- One-Click JSON Export: Copy complete record payloads directly to your clipboard.
5. Native File Icons
- Automatic custom geometric urn file icons for
.pithos and .midb container files in the VS Code File Explorer.
| Format |
Extension |
Container Superblock |
Description |
| Pithos Container v2 |
.pithos |
DIOGENES (v2) |
Single-file vector database container with Matryoshka bitsets, FP8 sidecars, and Arrow metadata. |
| MIDB Vector File |
.midb |
DIOGENES |
Matryoshka Index Database binary container. |
Extension Settings & Commands
Commands
Pithos: Open in Container Viewer (pithos.openViewer): Opens the selected .pithos or .midb file in the custom interactive inspector.
Custom Editor Association
This extension registers pithos.viewer as the default custom editor for *.pithos and *.midb files. Clicking on any container file in the File Explorer opens the viewer immediately.
Development & Building
Prerequisites
Build Instructions
# Install dependencies
npm install
cd webview-ui && npm install && cd ..
# Compile webview and extension bundles
npm run build
# Run binary reader and parser test suite
npm test
# Package VSIX for installation
npm run package
Debugging in VS Code
- Open this repository in VS Code.
- Press
F5 (or run Launch Extension) to start the Extension Development Host.
- Open any
.pithos file (such as test/fixtures/test_sample.pithos).
License
MIT License - Copyright (c) 2026 Finn Hertsch.
| |
