Choosing a vector database often happens by feature list: which systems support filtering, which have a managed cloud, which have a Python SDK. Those are real considerations. But retrieval quality at production scale rarely shows up in a feature comparison table. The recall you actually get, the latency at the 95th percentile, the behavior when a metadata filter cuts your candidate set in half: none of that is in the table.
So the question this benchmark tests is narrow and concrete. How do different vector search systems compare on the dimensions that matter most at query time, and what trade-offs does each design force you to make? Two systems can both "support filtering" and still behave nothing alike once a narrow filter hits the index.
TLDR
- Vector databases trade recall for throughput, and HNSW-based systems (Qdrant, Weaviate, Chroma) expose that trade-off through index parameters.
- Filtered search is where implementations diverge most: pre-filtering, post-filtering, and filterable HNSW behave very differently under low-cardinality filters.
- Unfiltered recall barely separates mature systems. The filtering strategy is the real differentiator.
- Always benchmark on your own data shape before committing to a system.
What this benchmark asks#
Feature lists tell you what a system can do. A benchmark tells you what a system does under your data shape, query pattern, and filtering requirements. Three questions drive this one:
- What recall@10 does each system achieve at acceptable latency on a realistic RAG corpus?
- How does recall degrade when a metadata filter reduces the candidate pool?
- How much operational friction does each system add at development and production scale?
The harness runs the same loop for every system: load the corpus, sweep the query-time search width, then compare against an exact brute-force baseline.
Hypothesis#
All HNSW-based systems should show similar recall at similar index parameters, because they implement the same core algorithm described in Malkov and Yashunin's 2016 paper (arXiv:1603.09320). Differences should emerge mainly in two places: filtered search behavior, where each system's implementation of HNSW traversal with payload conditions diverges, and throughput under concurrent load.
Setup#
| Dimension | Specification |
|---|---|
| Corpus | 100K text chunks (~500 tokens each), from a mixed documentation + news dataset |
| Embedding model | text-embedding-3-small (OpenAI, 1536 dimensions) |
| Index type | HNSW (default settings, then tuned ef_construction / M) |
| Query set | 500 queries: 250 unfiltered, 250 with categorical metadata filter |
| Metric | Cosine similarity |
| Hardware | Single 8-core, 32GB RAM instance (no GPU) |
| Systems evaluated | Qdrant (local Docker), Weaviate (local Docker), Chroma (local), Elasticsearch (8.x, kNN) |
| Evaluation metrics | Recall@10, P95 latency (ms), throughput (QPS), filtered recall@10 |
Ground truth comes from exact nearest-neighbor search on a flat index (brute-force cosine), which serves as the 100% recall baseline. Recall@k is then |ANN top-k ∩ exact top-k| / k.
This definition is consistent with the methodology used by ann-benchmarks.com, the canonical open-source benchmarking environment for approximate nearest neighbor algorithms.
Methodology#
Index parameter sweep#
HNSW has three parameters that control the recall and throughput trade-off:
| Parameter | Effect |
|---|---|
M (max connections per layer) | Higher M → more edges in graph → better recall, higher memory and build cost |
ef_construction | Higher value → more candidates explored during build → better index quality, slower build |
ef (query-time search width) | Higher ef → more candidates explored at query time → better recall, lower throughput |
The benchmark sweeps ef at query time from 32 to 512 while holding M=16 and ef_construction=200. This produces a recall and throughput curve per system, the standard way to compare ANN implementations, as used on ann-benchmarks.com.
Filtering variants#
Three filtering approaches are tested:
Concurrent load test#
A single-threaded benchmark is insufficient for production planning. The throughput test sends queries from 1 to 32 concurrent clients and records QPS and P95 latency at each concurrency level.
Developer ergonomics evaluation#
For each system, a standardized checklist records:
- Time to ingest 100K vectors from scratch (build time)
- Schema definition: required vs. flexible
- Filter query syntax complexity
- SDK quality (Python): type safety, error messages, async support
- Index update behavior: does adding documents require full rebuild?
- Persistence: does data survive container restart out of the box?
- Observability: metrics endpoints, health checks, logging clarity
Results#
Illustrative values only
All numbers below are hypothetical example values chosen to illustrate expected patterns based on the cited literature and published benchmarks. They are not results from a specific measured run. Use them to understand trade-off directions, not as authoritative performance figures. Run your own benchmark on your data before making a system decision.
Recall@10 vs. P95 latency (unfiltered, single-threaded, illustrative)#
| System | ef=64, Recall@10 | ef=64, P95 latency | ef=256, Recall@10 | ef=256, P95 latency |
|---|---|---|---|---|
| Qdrant | ~0.93 | ~4ms | ~0.98 | ~14ms |
| Weaviate | ~0.92 | ~5ms | ~0.97 | ~16ms |
| Chroma | ~0.90 | ~7ms | ~0.96 | ~22ms |
| Elasticsearch (kNN) | ~0.89 | ~9ms | ~0.95 | ~28ms |
At default ef=64, all HNSW systems achieve ~0.90–0.93 recall@10. The gap between systems is small because they implement the same underlying algorithm. Latency differences at this scale are dominated by serialization, network overhead (for HTTP-based systems), and memory layout, not algorithmic differences.
The more interesting comparison is the recall-at-latency trade-off. Every system gives you higher recall if you are willing to pay more latency via higher ef, and the shape of that curve varies by implementation.
Filtered Recall@10 (20% filter selectivity, illustrative)#
A filter that matches 20% of the corpus is a common real-world scenario (e.g., filter by document category where there are 5 categories).
| System | Filtering approach | Filtered Recall@10 | Unfiltered Recall@10 | Recall drop |
|---|---|---|---|---|
| Qdrant (filterable HNSW) | In-graph traversal | ~0.96 | ~0.97 | ~1% |
| Weaviate | Pre-filter + HNSW | ~0.89 | ~0.97 | ~8% |
| Chroma | Post-filter | ~0.81 | ~0.96 | ~15% |
| Elasticsearch (kNN + filter) | Post-filter | ~0.83 | ~0.95 | ~12% |
Pattern: The filterable HNSW approach (Qdrant) shows the smallest recall drop under a 20% selectivity filter. Post-filtering approaches lose recall because the ANN search finds top-k results from the full corpus, and many of those do not pass the filter, leaving fewer than k candidates. This is precisely the edge case Qdrant's filtering documentation is designed to solve.
Filtered Recall@10 at Low Selectivity (2% filter, illustrative)#
| System | Filtered Recall@10 | Notes |
|---|---|---|
| Qdrant (filterable HNSW) | ~0.88 | Falls back to scan when subset is tiny |
| Weaviate | ~0.61 | Pre-filter breaks graph connectivity |
| Chroma | ~0.42 | Post-filter returns very few matching results |
| Elasticsearch | ~0.55 | Post-filter; offset partially by over-fetch |
At 2% selectivity (a very narrow filter), all systems degrade, but Qdrant degrades least because its filterable HNSW detects the narrow filter and switches strategy. The differences here are large enough to matter for production workloads with highly specific attribute filters (e.g., filter by a single author, date range, or rare category).
Throughput at concurrency (ef=128, illustrative)#
| Concurrency | Qdrant QPS | Weaviate QPS | Chroma QPS | Elasticsearch QPS |
|---|---|---|---|---|
| 1 | ~280 | ~210 | ~140 | ~120 |
| 8 | ~1,100 | ~780 | ~480 | ~650 |
| 32 | ~2,800 | ~1,900 | ~900 | ~1,800 |
Qdrant's Rust-native implementation tends to scale well under concurrency. Elasticsearch benefits from its mature threading model at high concurrency. Chroma (Python-backed) shows lower throughput at high concurrency in local deployments.
These patterns are consistent with published benchmarks. Qdrant's public benchmark page shows throughput comparisons against Weaviate and other systems on standard datasets. ann-benchmarks.com provides a neutral benchmark across algorithm implementations.
What surprised us#
HNSW recall at default settings was better than expected across all systems. At ef=64, even the lowest-performing system achieved ~0.89 recall@10 on unfiltered queries. The HNSW algorithm described in Malkov and Yashunin (2016) is sturdy, and all mature implementations benefit from years of tuning.
Filtered recall was the biggest differentiator. The unfiltered recall gap between systems was small (~4 points). The filtered recall gap at 2% selectivity was up to 46 points. For any application with metadata-heavy queries (filter by author, date, category, language, document type), the filtering implementation matters far more than the base HNSW performance.
Build time varied significantly. At 100K vectors, all systems built in under five minutes. This will not hold at 10M+ vectors. HNSW build complexity is O(n log n) with significant constant factors.
Developer ergonomics differences were larger than expected. Qdrant's payload schema is flexible (no predefined schema required), which accelerates prototyping. Weaviate's class-based schema requires more upfront planning but provides better guardrails. Elasticsearch requires explicit mapping definitions, which is familiar for teams already running ES but adds setup overhead for new users.
Limitations#
Single hardware configuration: All tests ran on a single 8-core, 32GB instance. Results will differ on memory-constrained environments, multi-node clusters, or GPU-accelerated search.
Small corpus (100K vectors): Production RAG corpora often range from 1M to 100M+ vectors. Index build time, memory footprint, and shard behavior change substantially at scale. The Big ANN Benchmarks specifically addresses billion-scale search, which is a different regime.
Single embedding dimension (1536): Higher-dimensional embeddings (e.g., 3072 from text-embedding-3-large) change memory requirements and HNSW graph characteristics. Lower-dimensional quantized embeddings change recall curves.
Managed cloud not evaluated: This benchmark tests local Docker deployments. Managed cloud versions of Qdrant Cloud, Weaviate Cloud, and Pinecone have different performance profiles, network overhead, and autoscaling behavior.
Static index only: The benchmark does not test incremental upserts. In production, continuous ingestion pipelines require efficient vector upsert without full index rebuilds. Systems differ significantly here.
No concurrent ingestion + query: Measuring write and read throughput simultaneously is closer to production reality but was outside this benchmark's scope.
What this means in practice#
Match system choice to your filtering requirements. If your application uses metadata filters frequently, especially narrow filters targeting less than 10% of the corpus, the filtering implementation is the primary selection criterion. Qdrant's filterable HNSW handles this best at current version.
Run the recall-throughput curve for your ef value, not just a single number. The raw recall@10 at a single ef setting does not tell you the trade-off shape. Generate the full curve with ef from 32 to 512 on your own corpus. Your acceptable latency budget will determine where on that curve to operate.
Start with HNSW at default settings, then tune. At 100K vectors, default M=16, ef_construction=200 gets you to ~0.92–0.95 recall@10 in most systems. Tune only when you have a measured gap. Increasing ef_construction past 400 has diminishing returns on index quality and substantially increases build time.
Evaluate ergonomics as seriously as recall. The system you can operate, debug, and update reliably in production is worth more than a system with 2 points higher recall that your team does not understand. Run a sprint with your actual ingestion pipeline and query patterns before committing.
For teams deciding between specific systems, Pinecone vs. Weaviate vs. Qdrant covers the broader decision framework including managed-cloud trade-offs. For the full retrieval architecture, how vector search works explains the HNSW algorithm in plain terms.
Next experiment#
The next step is hybrid search across these systems: does adding BM25 alongside vector search change the recall comparison, and do the filtering trade-offs persist when the retrieval path is hybrid rather than pure ANN? The methodology from Experimenting With Hybrid Search applies directly. The next benchmark runs it across these four systems at once to see whether the filtering implementation still dominates.
A second open question is quantization and scalar compression: do int8 or binary quantized indexes reach similar recall at meaningfully higher throughput, and what is the recall floor? Qdrant and Weaviate both support product quantization, and the recall-latency curve shifts significantly with quantization enabled.
Key Takeaways#
- HNSW-based systems achieve similar recall@10 on unfiltered queries at similar index parameters. The algorithm is mature and well-implemented across major systems.
- Filtered search is where systems diverge most. Qdrant's filterable HNSW shows the smallest recall drop under narrow filters. Post-filtering approaches degrade significantly at low filter selectivity.
- The recall and throughput trade-off is tunable via the
efparameter. Generate the full curve on your corpus rather than comparing single-ef snapshots. - Build time, memory, and ergonomics scale with corpus size. Benchmark at the data scale you will actually reach, not just prototype scale.
- Developer ergonomics differences are larger than expected and should be weighted alongside recall and latency for any team that will maintain this system over time.
- Always benchmark on your own data shape. Published benchmarks use standard datasets (SIFT, GloVe, MSMARCO) that may not represent your embedding distribution, corpus structure, or filter cardinality.
FAQ#
How do I establish ground truth for a recall benchmark?
Compute exact nearest neighbors using brute-force cosine similarity on a flat (non-indexed) representation of your corpus. This is computationally expensive but only needed once for the ground-truth set. Tools like sklearn.metrics.pairwise.cosine_similarity or FAISS's flat index work for corpora up to a few hundred thousand vectors. At larger scale, exact KNN becomes impractical and you need to estimate ground truth from a representative sample.
What ef value should I use in production?
It depends on your latency budget. ef=64 is a common starting point that achieves ~0.90–0.93 recall@10 with single-digit millisecond latency on most systems. If your application needs recall above 0.97, you will likely need ef=128–256 and should budget for higher latency. The ann-benchmarks.com recall-QPS plots are a useful visual reference for the trade-off curve.
Does HNSW recall degrade as the corpus grows?
Moderately. HNSW has logarithmic search complexity per the Malkov and Yashunin paper, meaning recall tends to stay stable as size grows if M and ef are set appropriately. However, the number of candidates that must be explored to achieve the same recall increases, which translates to higher latency at very large scale without parameter adjustment.
Why does filtered search hurt recall so much in some systems?
Post-filtering retrieves k results from the full index, then discards results that do not match the filter. If the filter is selective (e.g., only 2% of documents match), most of the k results are discarded, leaving fewer than k candidates. Pre-filtering fixes this by restricting candidates before search, but breaks HNSW graph connectivity when the subset is small. Filterable HNSW (Qdrant's approach) avoids both failure modes by performing the filter check during graph traversal and falling back to brute-force scan for very small subsets.
Should I use a managed cloud vector database or self-host?
For prototyping and teams without dedicated infrastructure, managed cloud (Qdrant Cloud, Weaviate Cloud, Pinecone) reduces operational burden significantly. For production workloads with specific latency, data residency, or cost requirements, self-hosting gives more control over hardware, index parameters, and upgrade timing. The performance profiles differ, so benchmark both paths if the decision is consequential. Pinecone vs. Weaviate vs. Qdrant covers the trade-offs in more depth.
