RyanCodrai/turbovec: Zero-Training Vector Search with TurboQuant

RyanCodrai/turbovec: Zero-Training Vector Search with TurboQuant

Vector search doesn't have to mean choosing between memory efficiency and setup complexity. Most production-grade approximate nearest neighbor (ANN) indexes demand a training phase—k-means clustering, codebook optimization, parameter tuning—before you can query anything. That friction blocks real-time ingestion, complicates MLOps pipelines, and burns engineering hours on infrastructure instead of product. turbovec eliminates that trade-off entirely. Built on Google Research's TurboQuant algorithm and implemented in Rust with Python^{↗ Bright Coding Blog} bindings, it offers online vector indexing with no training step, no rebuilds as your corpus grows, and memory compression that drops a 10-million-document float32 corpus from 31 GB to 4 GB—while searching faster than FAISS on ARM and competitive on x86. For developers building RAG, semantic search, or recommendation systems where latency, privacy, and operational simplicity matter, turbovec is worth a serious look.

What is RyanCodrai/turbovec?

turbovec is an open-source vector index maintained by Ryan Codrai, distributed under the MIT License. The repository has accumulated 12,690 stars and 1,124 forks as of its last commit on June 10, 2026. Despite the Python badge as primary language, the computational core is written in Rust; Python bindings make it accessible to the ML/data science ecosystem that expects pip install workflows.

The project implements TurboQuant, a data-oblivious quantizer published by Google Research (ICLR 2026, arXiv:2504.19874). "Data-oblivious" means the quantization strategy is derived from mathematical properties of high-dimensional spaces—specifically, that random rotation makes coordinate distributions predictable via Beta/Gaussian asymptotics—rather than fitted to your specific dataset. This removes the classic vector database bootstrapping problem: with FAISS Product Quantization (PQ), you must accumulate enough vectors, run k-means++, and tune sub-quantizer counts before your index is usable. With turbovec, you call add() and the vector is queryable immediately.

The timing is relevant. The 2023-2025 explosion of retrieval-augmented generation (RAG) has pushed vector search from niche ML infrastructure into mainstream application backends. Developers who previously outsourced this to managed services (Pinecone, Weaviate Cloud, OpenAI's retrieval) are increasingly bringing search back on-premise for cost, latency, and data-sovereignty reasons. turbovec targets exactly that shift: pure local execution, no network calls, no vendor lock-in, with performance that rivals or beats the established C++ alternative.

Key Features

Online ingest with zero training. The defining characteristic. Vectors are indexed on arrival; there's no train() call, no k-means convergence, no parameter grid search for nlist or M sub-quantizers. The index grows incrementally without quality degradation or mandatory rebuilds. This simplifies streaming architectures and reduces time-to-first-query from potentially hours to milliseconds.

Aggressive memory compression via scalar quantization. TurboQuant uses Lloyd-Max scalar quantization on rotation-predicted coordinate distributions. At 2-bit precision, a 1536-dimensional float32 vector compresses from 6,144 bytes to 384 bytes—16× reduction. At 4-bit, it's 768 bytes (8×). The 31 GB → 4 GB claim for 10M documents assumes 2-bit storage plus overhead.

SIMD-optimized search kernels. Hand-written NEON intrinsics for ARM and AVX-512BW for x86_64 (with AVX2 fallback). On Apple M3 Max, turbovec outperforms FAISS IndexPQFastScan by 10–19% across all tested configurations. On Intel Sapphire Rapids, it wins 4-bit configs by up to ~5% and trails 2-bit by modest margins (most visibly ~8% at d=1536 single-threaded) where FAISS's AVX-512 VBMI path excels on short accumulate loops.

Native filtered search. Pass an allowlist of uint64 IDs or a slot bitmask directly to search(). The SIMD kernel short-circuits entire 32-vector blocks with zero allowed entries before any lookup-table work; individual non-allowed slots inside scored blocks are dropped at heap insertion. You get up to k results from exactly the allowed set—no over-fetching, no recall penalty for selective filters. This enables hybrid retrieval pipelines where a cheap filter (SQL, BM25, time range) narrows candidates before dense reranking.

Stable external IDs with deletion. IdMapIndex maps your uint64 IDs to internal slots, supporting O(1) removal by ID. Persistence preserves this mapping—reload and your IDs remain valid.

Framework drop-ins. Optional packages replace in-memory vector stores in LangChain, LlamaIndex, Haystack, and Agno without pipeline rewrites.

Use Cases

Privacy-critical RAG deployments. Healthcare, legal, financial services, and government applications often cannot send documents or embeddings to third-party APIs. turbovec runs entirely locally; pair with an open-source embedding model (BGE, E5, GTE) and you have a fully air-gapped retrieval stack. No data leaves your VPC, no subscription metering, no network latency on query.

Real-time ingestion pipelines. Recommendation systems, social feeds, and monitoring tools receive vectors continuously. Traditional PQ indexes require periodic retraining that creates version-management headaches or query downtime. turbovec's online add means your index is always current—no batch windows, no quality cliffs after retrain.

Memory-constrained edge and embedded devices. A 4 GB footprint for 10M documents at 2-bit makes semantic search viable on hardware where loading uncompressed float32 embeddings would be impossible. ARM optimization (the 10–19% speedup platform) aligns with edge deployment on Apple Silicon, AWS^{↗ Bright Coding Blog} Graviton, or mobile-class processors.

Hybrid search architectures. Use your existing relational database or text search engine for coarse filtering (tenant isolation, date ranges, keyword matches), then pass the candidate ID set to turbovec for dense semantic reranking. The kernel-level filter integration avoids the "fetch 1000, rerank 10" waste pattern that hurts latency and throughput in naive two-stage systems.

Rapid prototyping and evaluation. When you're comparing embedding models or chunking strategies, the last thing you want is index training overhead between experiments. turbovec lets you swap models, re-embed, and re-index in minutes rather than hours.

Installation & Setup

Python

pip install turbovec

This installs the pre-built wheel with Rust extensions compiled for your platform. For framework integrations, use extras:

pip install turbovec[langchain]      # LangChain replacement
pip install turbovec[llama-index]    # LlamaIndex replacement
pip install turbovec[haystack]       # Haystack replacement
pip install turbovec[agno]           # Agno replacement

Building from source (Python)

If you need a custom build or your platform lacks wheels:

pip install maturin
cd turbovec-python
maturin build --release
pip install target/wheels/*.whl

maturin compiles the Rust core and packages it into a Python wheel. The --release flag enables optimizations; expect compile times of several minutes depending on your Rust toolchain.

Rust

cargo add turbovec

Or add manually to Cargo.toml. All x86_64 builds target x86-64-v3 (AVX2 baseline, Haswell 2013+) via .cargo/config.toml. The AVX-512 kernel is runtime-gated via is_x86_feature_detected!—it activates automatically on compatible hardware without recompilation.

cargo build --release

Benchmark data (optional)

python3 benchmarks/download_data.py all            # all datasets
python3 benchmarks/download_data.py glove          # GloVe d=200
python3 benchmarks/download_data.py openai-1536    # OpenAI DBpedia d=1536
python3 benchmarks/download_data.py openai-3072    # OpenAI DBpedia d=3072

Real Code Examples

Basic vector indexing and search

The simplest turbovec workflow: create an index, add vectors, search, persist.

from turbovec import TurboQuantIndex

# Initialize: 1536 dimensions, 4-bit quantization
index = TurboQuantIndex(dim=1536, bit_width=4)

# Online ingest — no train() call needed
index.add(vectors)
index.add(more_vectors)  # incremental, no rebuild

# Search: returns scores and internal indices
scores, indices = index.search(query, k=10)

# Persistence
index.write("my_index.tv")
loaded = TurboQuantIndex.load("my_index.tv")

Note bit_width=4 trades memory for accuracy versus bit_width=2. The choice depends on your recall requirements and corpus size; see the benchmark charts in the repository for guidance.

Stable external IDs with deletion

When you need IDs that survive across sessions and support removal:

import numpy as np
from turbovec import IdMapIndex

index = IdMapIndex(dim=1536, bit_width=4)

# Your own uint64 IDs — from a database primary key, UUID hash, etc.
index.add_with_ids(vectors, np.array([1001, 1002, 1003], dtype=np.uint64))

scores, ids = index.search(query, k=10)  # returns your external IDs
index.remove(1002)                        # O(1) deletion by ID

index.write("my_index.tvim")
loaded = IdMapIndex.load("my_index.tvim")

IdMapIndex is essential for production systems where internal slot numbers would leak abstraction boundaries or complicate integration with existing data models.

Hybrid retrieval with allowlist filtering

import numpy as np
from turbovec import IdMapIndex

idx = IdMapIndex(dim=1536, bit_width=4)
idx.add_with_ids(vectors, ids)

# Stage 1: external system produces candidates
allowed = np.array(
    db.execute("SELECT id FROM docs WHERE tenant=?", (t,)).fetchall(),
    dtype=np.uint64
)

# Stage 2: dense rerank within exactly the candidate set
scores, ids = idx.search(query, k=10, allowlist=allowed)

The allowlist is honored inside the SIMD kernel at 32-vector block granularity. Selective filters avoid most computation rather than scoring everything and discarding—critical when your candidate set is 0.1% of the corpus. Output length is min(k, len(allowed)); no padded fallbacks when candidates are scarce.

Rust equivalent

use turbovec::TurboQuantIndex;

let mut index = TurboQuantIndex::new(1536, 4).unwrap();
index.add(&vectors);
let results = index.search(&queries, 10);
index.write("index.tv").unwrap();
let loaded = TurboQuantIndex::load("index.tv").unwrap();

The Rust API mirrors Python's structure with idiomatic error handling (Result types). For stable IDs, use IdMapIndex analogously.

Advanced Usage & Best Practices

Choose bit width for your recall budget. The README documents that both 2-bit and 4-bit reach R@≥0.997 by k=4 on OpenAI d=1536/3072 embeddings. At R@1, 4-bit leads 2-bit by 0.2–1.9 points. For user-facing search where top-1 accuracy matters, 4-bit is likely worth the 2× memory. For large-scale candidate generation where you rerank with a cross-encoder, 2-bit may suffice.

Enable TQ+ calibration for low-dimensional embeddings. The base TurboQuant algorithm assumes high-dimensional asymptotics; at d=200 (GloVe-style), coordinates drift from the ideal Beta distribution. TQ+ fits per-coordinate shift and scale during the first add(), mapping empirical quantiles to the canonical distribution. This is automatic when needed, but be aware that the first add incurs slightly more work. The calibration freezes afterward—subsequent adds reuse it.

Leverage length-renormalization. The ||v|| / ⟨u, x̂⟩ correction stored per vector removes downward bias in inner-product estimation from scalar quantization. This happens automatically; you don't configure it. But understand that it's most impactful at low bit widths where quantization shrinkage is largest—another reason 2-bit works better than naive analysis might suggest.

Profile your filter selectivity. The allowlist optimization shines when most 32-vector blocks can be skipped. If your filter passes >50% of vectors, you're approaching the unfiltered path's cost. Consider whether pre-filtering in your SQL/BM25 stage is pulling its weight, or if a post-filter with larger k followed by truncation might be simpler.

Test ARM vs. x86 for your workload. The 10–19% ARM speedup and x86 4-bit wins are medians over specific benchmarks. Your embedding dimension, batch size, and query distribution may shift the balance. The self-contained benchmark scripts in benchmarks/suite/ let you replicate on your hardware.

Comparison with Alternatives

FAISS remains the mature, broadly-deployed choice with extensive index types (HNSW, IVF, GPU) beyond PQ. turbovec doesn't replace it universally—it specifically wins on operational simplicity (no training), ARM optimization, and kernel-integrated filtering. Managed services eliminate infrastructure but introduce latency, cost scaling, and data residency concerns that turbovec avoids entirely. Your choice depends on whether "managed" is a convenience or a constraint.

FAQ

Does turbovec support GPU acceleration? No. The current implementation targets CPU SIMD (NEON, AVX-512BW/AVX2) only.

What's the minimum viable corpus size? There's no minimum. The online ingest works from the first vector; however, the TQ+ calibration step at d=200 benefits from a representative sample in the initial add().

Can I update vectors in-place? The README documents add, add_with_ids, and remove. In-place modification of existing vectors is not described; remove and re-add is the likely pattern.

Is the MIT license permissive for commercial use? Yes. MIT permits commercial use, modification, and distribution with attribution. See the repository's LICENSE file for full text.

How does recall compare to uncompressed (brute-force) search? Benchmarks show R@1 within ~1-2 points of FAISS PQ at comparable bit rates, reaching near-perfect recall by k=8 on high-dimensional embeddings. See the repository's benchmarks/results/ JSON files for exact numbers.

Does it support multi-vector (ColBERT-style) retrieval? Not documented. turbovec indexes single vectors per document.

What's the catch with "zero training"? The quantization quality depends on the mathematical assumption that random rotation produces predictable coordinate distributions. This holds well for high-dimensional embeddings (d≥768) and improves further with TQ+ calibration. Very low dimensions or highly structured non-semantic data may see larger distortion.

Conclusion

turbovec solves a specific, painful problem in vector search infrastructure: the training-phase bottleneck that complicates real-time ingestion and operationalizes PQ indexes. By grounding quantization in high-dimensional geometry rather than data-dependent optimization, it achieves competitive recall with dramatically simpler operations. The Rust core with Python bindings bridges performance and accessibility; kernel-integrated filtering and stable-ID support address production requirements that many research implementations ignore.

It's best suited for teams building self-hosted RAG or semantic search where ARM deployment, data privacy, or streaming ingestion are priorities. If you're already invested in FAISS GPU indexes or need HNSW-graph traversal for million-scale exact search, turbovec complements rather than replaces your stack. For everyone else—especially those who've been paying managed service premiums to avoid index training headaches—it's worth benchmarking on your workload.

Explore the repository, run the self-contained benchmarks, and see how the zero-training approach fits your pipeline: https://github.com/RyanCodrai/turbovec