Benchmarks
Numbers we measured, not numbers we hope for.
Every metric below comes from ./run.sh benchmark, a test suite that runs thereal app code (real FAISS bridge, real vector store, real embedding models), not mocks. Run it on your own machine; the command is in the repo.
| Metric | Value | What it means |
|---|---|---|
| vector.search_p50_ms | 0.11 ms | Semantic search, median (2,000-vector HNSW index) |
| vector.search_p95_ms | 0.46 ms | Semantic search, 95th percentile |
| vector.add_2000_ms | 157.7 ms | 2,000 entries added to the real FAISS store (0.08 ms/entry) |
| ram.vector_store_growth_mb | 8.08 MB | Footprint growth for 2,000 vectors + metadata |
| file.read_300_files_ms | 6.3 ms | 300 files read from disk (editor open-path proxy) |
| Embeddings (on-device, ANE) | 3.2 / 7.4 / 15.0 ms | bge-small (384d) / bge-base (768d) / bge-large (1024d), avg per embedding |
Baseline: M4 MacBook Pro, 2026-08-03, Debug build. Full methodology and regression gates:BENCHMARKS.md
Why these KPIs
The numbers our users actually care about
RAM: the founding story
Compass exists because Cursor + Docker ate a 16GB MacBook. Footprint and growth are tracked like bugs: 8 MB for 2,000 semantic entries, and we publish the delta because bloat is a bug.
Search latency: retrieval must feel instant
RAG quality is worthless if retrieval feels slow. Sub-millisecond at 2,000 entries; the target is sub-10ms at 1,000,000. The index grows; the latency shouldn't.
Embedding latency: the Neural Engine promise
All bundled models must stay under 100 ms per embedding. Current: 3–15 ms.
On the bench
Next metrics, in the open
FIM completion latency (p50/p95)
The <100ms claim, measured with the real 4B on-device model: time-to-first-token, prompt tokens/sec, generation tokens/sec, RSS before/after load. The metrics struct already exists in the harness; the run needs the model download and the stable merge. This is the marquee number; we won't publish it until we've actually measured it.
Startup-to-ready · index time · standalone app RAM
Launch → window usable; first + incremental index of a 100k-LOC repo; Release-build idle footprint. All queued, all published the moment they're measured.
Benchmarks run on every release and on demand. See theBenchmark workflow.
Don't trust us. Measure us.
Clone the repo, run ./run.sh benchmark, and compare on your own machine. That's the whole point.