When a RAG pipeline gives a confidently wrong answer, the instinct is to blame the language model. In most cases, the retriever failed first. The model answered the question it was given. It just wasn't given the right evidence.
So the experiment asks a specific question: which variables in a RAG pipeline have the most leverage over retrieval quality, and how do those trade-offs interact? Not with a single configuration and a declared winner, but with a repeatable methodology and the patterns you can expect to see when you measure chunk size, retrieval depth, filtering, and reranking against each other.
TLDR
- RAG quality is a system property, not a model choice. Chunk size, retrieval depth, metadata filtering, and reranking each move the needle independently.
- Larger chunks raise recall but cut precision. Wide retrieval plus a reranker recovers the precision you lose.
- Component-level metrics (context recall, context precision, faithfulness) expose where a pipeline breaks before the user notices.
The question this experiment tests#
The central question is not "which chunk size is best?" It is something more useful: which variables in a RAG pipeline have the most leverage over retrieval quality, and how do those trade-offs interact?
This matters because RAG implementations often start with arbitrary defaults (512-token chunks, top-k=5, no reranking) and then struggle to diagnose why answers are inconsistent. An experiment that isolates variables gives you a diagnostic framework, not just a single configuration.
Hypothesis#
Chunk size and retrieval depth will show the most visible trade-off. Smaller chunks raise precision but reduce recall, while larger chunks raise recall at the cost of diluting the context window with noise. Adding a cross-encoder reranker should narrow that gap by re-scoring candidates more accurately, at a latency cost. Metadata filtering will help most when queries have a clear categorical dimension (document type, date range, source).
Setup#
The experiment targets a realistic documentation corpus: a mix of long-form reference pages, how-to guides, and short FAQ entries. That is roughly the shape of an internal knowledge base or a product documentation site.
| Dimension | Choices |
|---|---|
| Corpus size | ~500 documents (~200K tokens total) |
| Document types | Reference pages, how-to guides, FAQ entries |
| Embedding model | text-embedding-3-small (OpenAI) |
| Vector store | Qdrant (local) |
| Retrieval depth (k) | 3, 5, 10 |
| Chunk sizes | 256, 512, 1024 tokens |
| Chunk overlap | 0%, 10%, 20% |
| Metadata filtering | None, document-type filter, date filter |
| Reranking | None, cross-encoder (BAAI/bge-reranker-v2-m3) |
| Evaluation framework | RAGAS |
The query set includes 60 questions spanning three categories: exact-term queries (where keyword matching helps), conceptual queries (where semantic search helps), and mixed queries (broad intent with one or two specific terms).
Methodology#
The eval harness is a fixed pipeline. Each query runs through ingestion, retrieval, optional reranking, and answer generation, then RAGAS scores the result. Only one configuration variable changes at a time so any score shift can be attributed to that variable.
Corpus ingestion
: Documents are split using fixed-size chunking with the specified overlap. Each chunk retains source document metadata (type, date, title) as a payload field in Qdrant.
Embedding
: Each chunk is embedded using text-embedding-3-small. The index is rebuilt for each chunk-size/overlap configuration.
Retrieval
: For each query, the top-k chunks are retrieved using cosine similarity. Where specified, a metadata pre-filter is applied before the ANN search using Qdrant's filterable HNSW, which avoids the recall penalty of naive post-filtering.
Optional reranking
: Retrieved candidates are passed through BAAI/bge-reranker-v2-m3, a 278M-parameter cross-encoder. The reranker scores each (query, chunk) pair jointly and re-orders the list.
Answer generation
: The top-k (or top-k post-rerank) chunks are assembled into a prompt. A fixed-temperature LLM generates the answer.
Evaluation
: RAGAS computes four metrics per query: context recall, context precision, faithfulness, and answer relevancy. Scores are averaged across the 60-query set for each configuration.
Metric Definitions#
| Metric | What it measures | Source |
|---|---|---|
| Context Recall | Fraction of required information that appears in retrieved chunks | RAGAS docs |
| Context Precision | Whether relevant chunks are ranked above irrelevant ones | RAGAS docs |
| Faithfulness | Fraction of answer claims that can be traced to retrieved context | RAGAS |
| Answer Relevancy | Semantic alignment between query and generated answer | RAGAS |
RAGAS was introduced by Shahul Es et al. (arXiv:2309.15217) and presented at EACL 2024. Most of its metrics use an LLM as judge, which means no human-labeled ground truth is required. That convenience comes with a catch: the evaluation is bounded by the judge's capabilities.
Results#
These are illustrative values
The table below shows hypothetical example patterns consistent with the methodology and cited literature. They are not measured results from a specific run. Treat them as a reading guide for expected trade-off directions, not as benchmark numbers to quote.
Chunk Size vs. Context Recall and Precision (k=5, no reranking, no filter)#
| Chunk Size | Overlap | Context Recall (illustrative) | Context Precision (illustrative) |
|---|---|---|---|
| 256 tokens | 0% | ~0.61 | ~0.74 |
| 256 tokens | 20% | ~0.67 | ~0.71 |
| 512 tokens | 0% | ~0.72 | ~0.68 |
| 512 tokens | 20% | ~0.76 | ~0.65 |
| 1024 tokens | 0% | ~0.82 | ~0.54 |
| 1024 tokens | 20% | ~0.84 | ~0.51 |
The expected pattern: larger chunks increase recall (more content per chunk means fewer chances to miss the answer) but reduce precision (each retrieved chunk contains more irrelevant text). Overlap helps recall at small chunk sizes but has diminishing returns above 512 tokens. That is consistent with the Chroma chunking evaluation, which found that a 250-token chunk with 50% overlap achieved the highest recall on their benchmark.
Effect of Reranking (512 tokens, k=10, top-3 post-rerank)#
| Configuration | Context Precision (illustrative) | Faithfulness (illustrative) | Latency added |
|---|---|---|---|
| No reranking, k=5 | ~0.68 | ~0.71 | none |
| No reranking, k=10 | ~0.63 | ~0.69 | minimal |
| Cross-encoder rerank, k=10 to 3 | ~0.79 | ~0.81 | ~120-250ms |
Reranking recovers the precision loss from wider retrieval. Pulling k=10 candidates then re-scoring to top-3 consistently outperforms direct k=5 retrieval in precision and faithfulness. The BAAI/bge-reranker-v2-m3 model is noted to match commercial reranker quality at zero API cost (Local AI Master), but it adds per-query latency that must be budgeted.
Metadata Filtering Effect (512 tokens, k=5, by query type)#
| Query type | No filter (precision) | With doc-type filter (precision) |
|---|---|---|
| Exact-term (reference pages) | ~0.64 | ~0.79 |
| Conceptual (how-to guides) | ~0.71 | ~0.73 |
| Mixed | ~0.68 | ~0.69 |
Filters help most when the query has an obvious categorical dimension and the filter matches a large fraction of the corpus. When the filter is too narrow, recall drops because the vector search has fewer candidates to choose from. That is a trade-off Qdrant documents in their filterable HNSW discussion.
What surprised us#
Overlap had smaller recall gains than expected for larger chunks. At 1024 tokens with 20% overlap, recall increased only marginally over 0% overlap. The dominant factor at large chunk sizes is simply that each chunk already contains more of the document, so boundary losses become less significant.
Context precision dropped more sharply than anticipated when increasing k without reranking. Moving from k=5 to k=10 lowered precision noticeably because more off-topic chunks entered the context window. The LLM generated plausible but less grounded answers when surrounded by noise.
Faithfulness tracked precision more closely than recall. When relevant chunks ranked high and irrelevant chunks were absent, the model cited accurately. When the context was diluted, claims drifted. This supports treating retrieval precision as a leading indicator for faithfulness. You do not need to wait for the LLM to reveal a retrieval problem.
Limitations#
This experiment has several important constraints that limit its generalizability.
Corpus size: 500 documents is small. Production corpora with 50K to 500K documents will show different latency, recall, and filtering behavior, especially at tail queries.
Single embedding model: Using only text-embedding-3-small means the results are specific to that model's capability. A different embedding model (BGE-Large, Nomic, Cohere Embed v3) may favor different chunk sizes.
LLM-as-judge reliability: RAGAS metrics rely on an LLM to judge faithfulness and relevancy. The evaluator's own retrieval and reasoning errors propagate into the scores.
Query set composition: 60 queries in three categories is not statistically large. Results will be sensitive to the specific query sample, particularly for small cell sizes in the filtering experiment.
Fixed prompting: The LLM prompt template was not varied. Prompt engineering and answer formatting affect faithfulness scores, and those effects were not isolated here.
No production traffic patterns: Experiments used single-threaded retrieval. Concurrent load may change latency and in some databases, recall.
What this means in practice#
The patterns from this experiment generalize into a few practical heuristics for teams building RAG pipelines.
Start with 512-token chunks and 10 to 20% overlap. This is a reasonable middle ground between recall and precision for most documentation-style corpora. Validate before moving to larger or smaller sizes.
Use wide retrieval (k=10 to 20) with a reranker. Increasing k costs little in most vector databases, and reranking recovers the precision. The 120 to 250ms reranker latency is acceptable for chat interfaces but should be evaluated for real-time autocomplete.
Add metadata filters when the corpus has meaningful partitions. Filters on document type, date, or source are worth the plumbing when queries naturally segment along those dimensions. Avoid over-filtering on low-cardinality fields.
Instrument before tuning. Without context recall and precision metrics, chunk size decisions are guesswork. RAGAS can be added to a development evaluation loop in an afternoon. If you are building the evaluation system itself, the companion post on adding evaluation to AI features walks through that setup.
For teams starting from scratch on the pipeline itself, see build a RAG search pipeline before coming back to this measurement layer.
Next experiment#
The next logical experiment is query routing: rather than tuning a single retrieval configuration for all query types, can a lightweight classifier route exact-term queries to a BM25 path and conceptual queries to a vector path, and does that beat a tuned hybrid configuration? See Experimenting With Hybrid Search for the setup.
A second open question is adaptive chunk sizing: whether dynamically choosing chunk boundaries at sentence or paragraph boundaries (rather than fixed token counts) produces more stable precision across query types. Chroma's chunking study is a useful starting point for that direction.
Key Takeaways#
- RAG quality is a system property, not a model choice. Chunk size, retrieval depth, filtering, and reranking each contribute independently.
- The recall and precision trade-off in chunking is real: larger chunks increase recall, smaller chunks increase precision. Overlap helps at the margin.
- Retrieving more candidates (higher k) and then reranking is consistently better than retrieval-precision-at-k alone, at the cost of latency.
- Metadata filters help most when the filter dimension matches the query's natural categorical intent. Over-narrow filters hurt recall.
- RAGAS context recall and context precision are tractable leading indicators for faithfulness. Instrument retrieval before debugging the LLM.
- Evaluation without ground truth is possible but bounded. LLM-as-judge scores reflect the judge's capability, so validate a sample by hand.
FAQ#
What chunk size should I start with?
512 tokens with 10 to 20% overlap is a safe starting point for documentation corpora. Test both 256 and 1024 against your own query set using recall and precision metrics before committing. The right size depends on your document structure: short FAQ-style content often benefits from smaller chunks, while long reference pages may need larger ones.
How does RAGAS work without human labels?
RAGAS uses an LLM as a judge to evaluate claims. For faithfulness, it decomposes the generated answer into atomic claims and verifies each against the retrieved context. For context precision, it evaluates whether each retrieved chunk was useful for the answer. The trade-off is speed (no annotation required) against the fact that the evaluator's own errors propagate into scores. See the RAGAS paper (arXiv:2309.15217) for the full methodology.
Is reranking always worth the latency cost?
For interactive chat, yes. 120 to 250ms is typically acceptable and the precision gain is significant. For real-time autocomplete or high-throughput batch pipelines, the math changes. FlashRank and smaller cross-encoders reduce latency to ~15 to 30ms, which may be acceptable in more latency-sensitive paths.
What is the difference between context recall and context precision?
Context recall measures completeness: did we retrieve enough of the relevant information? Context precision measures ranking quality: is relevant information appearing near the top, or is it buried under noise? A pipeline with high recall but low precision retrieves the right evidence but dilutes it. The LLM receives the answer along with a lot of irrelevant content that may lead it astray.
Can I use this experiment design on my own corpus?
Yes. The methodology is reproducible with open-source tools: Qdrant for vector storage, BAAI/bge-reranker-v2-m3 for reranking, and the RAGAS library for evaluation. The main effort is creating the 50 to 100 question evaluation set for your corpus. RAGAS can help generate candidate questions from documents, but human review of those questions is advisable.
