RAG From Development to Post-Deployment Monitoring, Part 1: Building the Pipeline

Your RAG demo worked. Responses were sharp, context was relevant, and the stakeholder presentation went well. Three weeks into production, users are complaining that the system confidently answers questions about documents it was never given, retrieves the wrong section of a long PDF, and occasionally ignores perfectly relevant content. None of that happened during the demo.

This gap is not a coincidence. Demo RAG is easy because the documents are small, the questions are scripted, and quality is judged by eyeball. Production RAG is hard because documents are messy, questions are unpredictable, and failure is often invisible until a user notices. Every layer of the pipeline (ingestion, chunking, embedding, indexing, retrieval, reranking, generation) compounds the ones above it. A 20% precision problem at retrieval becomes a 20% answer quality problem before the LLM even starts.

This series walks every stage of that pipeline from development through deployment and into ongoing monitoring. Part 1 covers everything you need to build the pipeline correctly before it goes live.

The RAG Mental Model#

Retrieval-Augmented Generation was introduced by Lewis et al. (NeurIPS 2020, arXiv:2005.11401) as a way to give language models access to external knowledge without retraining. The core idea is simple: instead of hoping the model memorized the right fact, you find the fact at query time and hand it to the model as context.

The three-step loop is:

1. Retrieve: given a user query, find the most relevant text passages from your knowledge base.
2. Augment: attach those passages to the prompt as context.
3. Generate: let the LLM produce a grounded answer from the combined input.

That loop is the whole system. Everything else (chunking, embedding, indexing, reranking) is infrastructure that makes step 1 work accurately at scale.

RAG vs Fine-Tuning#

The short version: fine-tuning encodes knowledge into model weights, which is expensive, slow to update, and hard to audit. RAG keeps knowledge outside the model, which is cheap to update, easy to version, and more transparent. We cover this tradeoff in full detail in Fine-Tuning vs RAG for AI Products. For most teams with frequently updated document collections, RAG is the right default.

Data Ingestion#

Before you can retrieve anything, you need to get your documents into a shape the pipeline can work with. Ingestion is the unglamorous part of RAG, but it is where most production problems originate. The flow moves from raw files to indexed vectors in a fixed sequence.

Sources and Parsing#

Documents come in formats that were designed for humans, not machines. PDFs encode layout through coordinates rather than semantic structure. HTML is mixed with navigation, scripts, and boilerplate. Office documents embed content in XML. Each format needs a different extraction strategy.

Common parsing tools:

• PDFs: PyMuPDF (fast, preserves layout metadata), pdfplumber (better table extraction), unstructured.io (handles mixed content).
• HTML/web: Trafilatura (best at body extraction, removes boilerplate), BeautifulSoup (flexible, requires more configuration).
• Office (DOCX/XLSX): python-docx for Word, openpyxl for Excel, or unstructured.io for unified handling.

For a detailed ingestion pipeline covering source connectors, parsers, and deduplication, see Build a Document Ingestion Pipeline.

Cleaning and Normalization#

Raw extracted text almost always contains noise. Headers and footers repeat on every PDF page. Tables extracted as plain text lose their relational structure. OCR output has character substitution errors. HTML extraction misses or double-counts content.

Minimum cleaning steps before chunking:

import re

def clean_text(raw: str) -> str:
    # Collapse excessive whitespace
    text = re.sub(r'\n{3,}', '\n\n', raw)
    text = re.sub(r'[ \t]+', ' ', text)
    # Strip page headers/footers that repeat
    text = re.sub(r'Page \d+ of \d+', '', text)
    # Remove null bytes and control characters
    text = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', text)
    return text.strip()

Metadata Extraction#

Every chunk must carry metadata. Metadata is what lets retrieval be precise rather than just similar. Without metadata, you cannot filter by document, date, author, or topic. You cannot delete stale content. You cannot attribute answers. You cannot debug retrieval failures.

Minimum metadata fields per chunk:

Extract section headings from the document structure before chunking. They give the retrieval system signal that similarity alone cannot provide.

Chunking Strategies#

Chunking is the single decision with the most leverage over retrieval quality. A chunk is the unit of retrieval: what gets embedded, indexed, stored, and returned. Get it wrong and you will retrieve context that is either too broad (noisy) or too narrow (incomplete).

Fixed-Size Chunking#

Split by character count. Simple and fast. Works poorly at boundaries because it cuts through sentences and paragraphs.

def fixed_chunk(text: str, size: int = 512, overlap: int = 64) -> list[str]:
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start += size - overlap
    return chunks

Use fixed chunking only for quick prototypes. It is not a production strategy.

Recursive Character Chunking#

Try progressively smaller separators (\n\n, \n, ., ) until the chunk fits. This preserves paragraph and sentence boundaries as much as possible. LangChain's RecursiveCharacterTextSplitter implements this pattern (LangChain splitter docs). Good default for most text corpora.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(text)

Sentence-Window Chunking#

Embed individual sentences, but at retrieval time return the surrounding window (e.g., ±2 sentences). This improves embedding precision because sentences are semantically atomic, while retrieval returns enough context for the LLM to reason from. LlamaIndex implements this as SentenceWindowNodeParser (LlamaIndex docs).

Semantic Chunking#

Use an embedding model to detect where semantic similarity drops, and split at those boundaries. This keeps topically coherent content together, regardless of word count. LangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser both implement this. Slower than recursive chunking (requires embedding during ingestion), but produces more semantically coherent chunks.

Parent-Document Retrieval#

Store both small chunks (for retrieval precision) and large parent chunks (for LLM context richness). Embed the small chunks, retrieve them by similarity, then fetch their parent chunks to pass to the model. LangChain's ParentDocumentRetriever implements this pattern (LangChain ParentDocumentRetriever). This is one of the most effective strategies for long documents.

Layout-Aware Chunking#

Use document structure (headings, sections, tables, lists) as the chunk boundary, not character count. Tools like unstructured.io extract structural elements and let you chunk by element type. Essential for technical documentation, legal contracts, and financial reports where structure carries meaning.

Chunk Size vs Overlap Tradeoffs#

Overlap of 10–15% of chunk size is a good starting point. More overlap improves boundary recall at the cost of storage and slight retrieval duplication.

Embeddings#

An embedding model converts text into a dense vector, a list of floating-point numbers that encodes semantic meaning. Similarity between two texts becomes distance between two vectors. The quality of your embedding model determines the ceiling of your retrieval quality.

Model Choices#

The MTEB Leaderboard#

The Massive Text Embedding Benchmark (MTEB) is the standard evaluation for embedding models across retrieval, classification, clustering, and reranking tasks (arXiv:2210.07316). The live leaderboard at huggingface.co/spaces/mteb/leaderboard ranks hundreds of models. Always check the retrieval task score specifically. The overall MTEB score includes tasks that may not match your use case.

Dimensions and Matryoshka Representation Learning#

OpenAI's third-generation embedding models support Matryoshka Representation Learning (MRL), which trains models such that the first N dimensions of the embedding are still meaningful on their own (OpenAI embeddings docs). You can truncate text-embedding-3-small from 1,536 to 512 dimensions with modest quality loss and significantly lower storage and compute costs.

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    input="Your text here",
    model="text-embedding-3-small",
    dimensions=512,  # MRL truncation
)
embedding = response.data[0].embedding  # 512-dimensional

Use the same dimensions value at both ingestion and query time. Mixing dimension sizes breaks retrieval.

Data Schemas#

A production RAG system needs a clear data model before any data is inserted. Documents have IDs. Chunks have IDs and parent document IDs. Metadata is queryable. Versions are tracked. Source of truth is declared.

Example Schema (SQL + JSON)#

-- Documents table: one row per source document
CREATE TABLE rag_documents (
    id            UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    source_path   TEXT NOT NULL UNIQUE,
    title         TEXT,
    content_hash  BYTEA,           -- SHA-256 of raw content, for change detection
    version       INT NOT NULL DEFAULT 1,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    updated_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    metadata      JSONB           -- flexible extra fields
);

-- Chunks table: one row per chunk, references a document
CREATE TABLE rag_chunks (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id     UUID NOT NULL REFERENCES rag_documents(id) ON DELETE CASCADE,
    chunk_index     INT NOT NULL,
    content         TEXT NOT NULL,
    content_hash    BYTEA,
    section_heading TEXT,
    page_number     INT,
    embedding       vector(1536),  -- dimension must match your model
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    metadata        JSONB,
    UNIQUE (document_id, chunk_index)
);

CREATE INDEX ON rag_chunks USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 64);

CREATE INDEX ON rag_chunks (document_id);
CREATE INDEX ON rag_chunks USING gin (metadata);

The JSONB metadata column lets you add domain-specific fields (author, topic, date range, classification level) without schema migrations. Use gin index on metadata for fast filtering.

The equivalent document record as JSON:

{
  "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
  "source_path": "contracts/msa_v3.pdf",
  "title": "Master Service Agreement v3",
  "version": 3,
  "metadata": {
    "department": "legal",
    "effective_date": "2026-01-01",
    "classification": "internal"
  }
}

Versioning and Source of Truth#

When a document changes, do not delete and re-insert blindly. Compute the content hash, compare it to the stored hash, and re-ingest only if the content changed. Keep the rag_documents table as the source of truth. Chunks are derived. They can always be regenerated from documents.

Deletion uses the ON DELETE CASCADE relationship:

-- Remove a document and all its chunks atomically
DELETE FROM rag_documents WHERE source_path = 'contracts/msa_v3.pdf';

Vector Database Choices#

pgvector runs inside PostgreSQL, which is the right starting point for most teams. As your index grows or your query patterns become more complex, purpose-built vector databases offer specialized features. Here is a practical comparison:

For a deeper comparison with decision criteria, see Pinecone vs Weaviate vs Qdrant.

Combining Postgres / MongoDB with a Vector DB#

One of the most common architecture questions in RAG: should the vector store live inside your primary database, or should it be a separate service?

Single-Store: pgvector in PostgreSQL#

Keep vectors and relational metadata in the same Postgres instance. Queries are a single join. Transactions are ACID. No sync required. pgvector's HNSW and IVFFlat indexes are mature and well-documented (pgvector GitHub).

-- Hybrid query: metadata filter + vector similarity in one statement
SELECT c.id, c.content, c.section_heading,
       1 - (c.embedding <=> $1::vector) AS similarity
FROM rag_chunks c
JOIN rag_documents d ON c.document_id = d.id
WHERE d.metadata->>'department' = 'legal'
  AND d.metadata->>'classification' = 'internal'
ORDER BY c.embedding <=> $1::vector
LIMIT 10;

Single-Store: MongoDB Atlas Vector Search#

MongoDB Atlas Vector Search lets you store vectors as fields in standard MongoDB documents, with HNSW-based approximate nearest-neighbor search (MongoDB Atlas Vector Search docs). This works well when your existing data is already in MongoDB and you want to add vector search without a separate service.

// Atlas Search index definition (JSON)
{
  "fields": [
    {
      "type": "vector",
      "path": "embedding",
      "numDimensions": 1536,
      "similarity": "cosine"
    },
    {
      "type": "filter",
      "path": "metadata.department"
    }
  ]
}

Separate Vector DB + Relational/Document Store#

When you need features that pgvector or MongoDB Atlas do not offer (native hybrid search, advanced payload filtering, very large scale), run a dedicated vector database (Qdrant, Weaviate, Pinecone) alongside your primary data store.

Sync strategies for the dual-write pattern:

The ID mapping is the critical piece. Every chunk in the vector database must have an ID that maps back to its row in the relational/document store. Store the primary database ID as the vector's payload ID, not a generated vector-store ID. This makes enrichment, deletion, and auditing straightforward.

# Store the Postgres chunk ID as the Qdrant point ID
qdrant_client.upsert(
    collection_name="rag_chunks",
    points=[
        PointStruct(
            id=str(chunk_row["id"]),   # Postgres UUID as string
            vector=embedding,
            payload={
                "document_id": str(chunk_row["document_id"]),
                "section_heading": chunk_row["section_heading"],
                "metadata": chunk_row["metadata"],
            }
        )
    ]
)

Database and Index Optimization#

Choosing the right index type and parameters is critical for both retrieval quality and cost. Both HNSW and IVFFlat are approximate nearest-neighbor algorithms. They trade a small amount of recall for dramatically faster search compared to exact k-NN.

HNSW vs IVFFlat#

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph where each node connects to its approximate nearest neighbors. Queries navigate the graph from coarse to fine. The original algorithm is described in arXiv:1603.09320 (Malkov & Yashunin, 2016).

IVFFlat (Inverted File with Flat quantization) partitions the vector space into lists clusters. At query time, it searches the probes nearest clusters. Faster to build than HNSW, lower memory, but lower recall at the same query speed.

Set ef_search at query time in pgvector:

SET hnsw.ef_search = 100;

SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity
FROM rag_chunks
ORDER BY embedding <=> $1::vector
LIMIT 10;

Pre-Filtering vs Post-Filtering#

Pre-filtering narrows the candidate set before ANN search. Post-filtering runs ANN search first, then applies metadata filters to the results. Pre-filtering is safer for recall: you never return filtered-out results. Post-filtering is faster but can produce fewer results than top_k if many results are filtered out.

Qdrant and Weaviate both support efficient pre-filtering by building separate payload indexes that the HNSW search can respect during traversal (Qdrant filtering docs, Weaviate filtering docs). In pgvector, use a SQL WHERE clause which the query planner integrates with the index scan.

Index Maintenance#

HNSW indexes in pgvector require no training data upfront (unlike IVFFlat, which needs to see data before it can partition). However, frequent inserts to a large HNSW index can degrade performance over time. For write-heavy workloads, consider periodic REINDEX during low-traffic windows:

REINDEX INDEX CONCURRENTLY idx_rag_chunks_embedding;

IVFFlat indexes need to be rebuilt if the data distribution shifts significantly from when the lists were created. The pgvector README recommends rebuilding after the dataset doubles (pgvector GitHub).

Cost Cutting#

Embedding costs and vector storage are the two main cost levers in RAG. Both are controllable without sacrificing retrieval quality significantly.

Embedding Caching#

Identical text should never be embedded twice. Cache embeddings by the SHA-256 hash of the normalized chunk text. This is especially valuable for documents that change partially, since only new or modified chunks need re-embedding.

import hashlib
import json

def get_or_create_embedding(text: str, cache: dict, model: str) -> list[float]:
    key = hashlib.sha256(text.encode()).hexdigest()
    if key in cache:
        return cache[key]
    embedding = create_embedding(text, model)
    cache[key] = embedding
    return embedding

In production, use a persistent cache (Redis or a database table) keyed by content hash and model version.

Batching#

OpenAI's embeddings API accepts up to 2,048 inputs per request (OpenAI API reference). Batching reduces per-request overhead and helps stay within rate limits:

async def embed_in_batches(texts: list[str], batch_size: int = 500) -> list[list[float]]:
    results = []
    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        resp = await client.embeddings.create(model="text-embedding-3-small", input=batch)
        results.extend(item.embedding for item in sorted(resp.data, key=lambda x: x.index))
    return results

Dimensionality Reduction (Matryoshka/MRL)#

As described in the embeddings section, MRL-trained models like text-embedding-3-small can be truncated. Moving from 1,536 to 512 dimensions reduces storage by 67% and speeds up distance computation. OpenAI reports that text-embedding-3-small at 512 dimensions outperforms text-embedding-ada-002 at 1,536 dimensions on MTEB (OpenAI embeddings announcement).

Scalar and Binary Quantization#

Vector databases support quantization to reduce storage and speed up search:

• Scalar quantization (SQ8): Compress each 32-bit float to an 8-bit integer. ~4x storage reduction with ~1% recall loss. Supported by Qdrant (Qdrant quantization docs), Weaviate (Weaviate quantization docs), and pgvector (via halfvec type for FP16).
• Binary quantization (BQ): Compress each dimension to a single bit. ~32x storage reduction. Works well for high-dimensional models (1,536+) with rescoring. Supported by Qdrant and Weaviate.

-- pgvector halfvec: 16-bit floats, ~2x storage reduction
CREATE TABLE rag_chunks_compressed (
    id UUID PRIMARY KEY,
    embedding halfvec(1536)
);

Model Right-Sizing#

text-embedding-3-small at $0.02/million tokens is the right default for most use cases. text-embedding-3-large costs $0.13/million tokens (6.5x more) and is only justified for quality-sensitive applications where the MTEB improvement translates to your specific query distribution. Open-source models like BGE-large eliminate API cost entirely at the price of GPU infrastructure.

Retrieval#

With documents ingested and indexed, retrieval is where the pipeline either earns its quality or exposes its weaknesses. Pure dense vector search is not the best strategy for most production applications. A production query path fans out across dense and sparse search, fuses the results, reranks, and only then generates.

Dense Vector Search#

Query the vector index for the top-k semantically similar chunks. Effective when the query and document use different words to express the same concept ("heart attack" vs "myocardial infarction"). Weak when the query contains specific identifiers (product codes, version numbers, proper nouns) that have no semantic neighbors.

async def dense_search(query: str, top_k: int = 20) -> list[dict]:
    q_emb = (await client.embeddings.create(
        model="text-embedding-3-small", input=[query]
    )).data[0].embedding

    with conn.cursor() as cur:
        cur.execute("""
            SELECT id, content, metadata,
                   1 - (embedding <=> %s::vector) AS score
            FROM rag_chunks
            ORDER BY embedding <=> %s::vector
            LIMIT %s
        """, (q_emb, q_emb, top_k))
        return [
            {"id": r[0], "content": r[1], "metadata": r[2], "score": r[3]}
            for r in cur.fetchall()
        ]

Sparse / Keyword Search (BM25)#

BM25 (Best Match 25) is a probabilistic keyword ranking function derived from TF-IDF. It handles exact keyword matches, rare terms, and specific identifiers far better than dense search. BM25 is described in Robertson & Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond" (Foundations and Trends in Information Retrieval, 2009).

In PostgreSQL, use tsvector and tsquery for BM25-style full-text search:

-- Add a tsvector column for full-text search
ALTER TABLE rag_chunks ADD COLUMN content_tsv tsvector
    GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;

CREATE INDEX ON rag_chunks USING gin (content_tsv);

-- BM25-style keyword search using ts_rank_cd
SELECT id, content,
       ts_rank_cd(content_tsv, plainto_tsquery('english', $1)) AS score
FROM rag_chunks
WHERE content_tsv @@ plainto_tsquery('english', $1)
ORDER BY score DESC
LIMIT 20;

For native BM25 in dedicated vector stores, Weaviate and Qdrant both have built-in sparse vector support.

Hybrid Search with Reciprocal Rank Fusion#

Hybrid search combines dense and sparse results. The standard fusion algorithm is Reciprocal Rank Fusion (RRF), introduced by Cormack et al. (SIGIR 2009). RRF avoids the need to normalize scores across different retrieval systems:

def reciprocal_rank_fusion(
    result_lists: list[list[dict]],
    k: int = 60
) -> list[dict]:
    """
    result_lists: multiple ranked result lists, each item has 'id' and optionally 'content'.
    k: constant (60 is the standard default from Cormack et al.).
    """
    scores: dict[str, float] = {}
    items: dict[str, dict] = {}

    for results in result_lists:
        for rank, item in enumerate(results):
            doc_id = str(item["id"])
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
            items[doc_id] = item

    ranked = sorted(scores.keys(), key=lambda x: scores[x], reverse=True)
    return [{"score": scores[doc_id], **items[doc_id]} for doc_id in ranked]

# Usage
dense_results = await dense_search(query, top_k=20)
sparse_results = await keyword_search(query, top_k=20)
hybrid_results = reciprocal_rank_fusion([dense_results, sparse_results])

Metadata Filtering#

Apply metadata filters before or during ANN search to narrow the candidate set. Filtering on document type, date range, classification level, or department dramatically improves precision for narrow queries without changing the underlying retrieval algorithm.

# pgvector: filter before ordering by vector distance
SELECT id, content, 1 - (embedding <=> %s::vector) AS score
FROM rag_chunks c
JOIN rag_documents d ON c.document_id = d.id
WHERE d.metadata->>'department' = 'engineering'
  AND (d.metadata->>'effective_date')::date >= '2025-01-01'
ORDER BY c.embedding <=> %s::vector
LIMIT 10;

Multi-Query and HyDE#

Single queries are a narrow lens. Two strategies expand coverage:

Multi-query: Generate multiple phrasings of the same question, retrieve for each, then deduplicate and merge results. LangChain's MultiQueryRetriever implements this automatically.

HyDE (Hypothetical Document Embeddings): Ask the LLM to generate a hypothetical answer to the query, then embed that hypothetical answer and use it for retrieval. The intuition is that a hypothetical document lives in a similar embedding space as real documents (arXiv:2212.10496). Particularly effective when queries are short and documents are long.

async def hyde_retrieve(query: str, top_k: int = 10) -> list[dict]:
    # Step 1: Generate a hypothetical answer
    hyp_response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Write a detailed paragraph that directly answers the question."},
            {"role": "user", "content": query}
        ],
        temperature=0.7,
    )
    hypothetical_doc = hyp_response.choices[0].message.content

    # Step 2: Embed the hypothetical answer and retrieve
    return await dense_search(hypothetical_doc, top_k=top_k)

Reranking#

Dense and sparse retrieval return candidates, not the final context. Reranking uses a more expensive cross-encoder model to re-score each candidate against the query, then selects the top results. Reranking consistently improves answer quality at a small latency cost, and is one of the highest-leverage improvements you can make to a retrieval pipeline.

A cross-encoder takes a (query, document) pair as joint input and outputs a relevance score. Unlike bi-encoders (which embed query and document independently), cross-encoders see both inputs simultaneously, which allows much richer interaction. The trade-off is speed: you cannot pre-compute cross-encoder scores.

Reranking Options#

Using Cohere Rerank#

import cohere

co = cohere.Client(api_key="YOUR_COHERE_API_KEY")

def cohere_rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    """
    Rerank a list of candidate chunks using Cohere's reranker.
    candidates: list of dicts with 'content' and 'id' fields.
    """
    response = co.rerank(
        query=query,
        documents=[c["content"] for c in candidates],
        top_n=top_n,
        model="rerank-english-v3.0",   # Cohere Rerank v3
    )
    return [
        {**candidates[r.index], "rerank_score": r.relevance_score}
        for r in response.results
    ]

See Cohere Rerank documentation for full API reference and model options.

Open-Source Cross-Encoder (sentence-transformers)#

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-large", max_length=512)

def cross_encoder_rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    pairs = [(query, c["content"]) for c in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(
        zip(candidates, scores),
        key=lambda x: x[1],
        reverse=True
    )
    return [
        {**item, "rerank_score": float(score)}
        for item, score in ranked[:top_n]
    ]

MMR for Diversity#

Maximum Marginal Relevance (MMR) reduces redundancy by penalizing candidates that are too similar to already-selected results. Useful when top retrieved chunks contain near-duplicate content from a long document. LangChain's MMRRetriever implements this with a configurable lambda_mult parameter (LangChain MMR docs).

When to Use What#

Query Optimization and Transformation#

The query that the user types is often a poor retrieval signal. Query transformation addresses this before vectors are even compared.

Query rewriting: Use an LLM to rephrase the query for better retrieval. Especially useful for conversational queries that contain pronouns ("what does it do?" → "what does [product name] do?") or for expanding abbreviations.

Query decomposition: Break a complex question into sub-questions, retrieve for each, and synthesize answers. LlamaIndex's SubQuestionQueryEngine and LangChain's query decomposition chain both implement this.

Query routing: Classify the query and route it to the most appropriate retrieval index or strategy. A question about financial data routes to the finance documents index. A question about product features routes to the product knowledge base.

HyDE: As described above, generate a hypothetical answer, then retrieve based on the hypothetical. Cross-reference: arXiv:2212.10496.

LLM Choice and Generation#

With reranked context in hand, generation is the final step. The choice of LLM and prompt construction significantly affects answer quality and cost.

Context Window and Cost Tradeoffs#

Prompt Construction#

Structure your prompt to make grounding explicit. Include source attribution in the context block so the model can cite sources in its answer.

async def generate_answer(query: str, reranked_chunks: list[dict]) -> str:
    # Format each chunk with its source for inline citation
    context_parts = []
    for i, chunk in enumerate(reranked_chunks):
        source = chunk.get("metadata", {}).get("source_path", "unknown")
        context_parts.append(f"[{i+1}] Source: {source}\n{chunk['content']}")

    context_text = "\n\n---\n\n".join(context_parts)

    messages = [
        {
            "role": "system",
            "content": (
                "You are a precise assistant. Answer the question using ONLY the "
                "provided context. Cite your sources using [1], [2], etc. "
                "If the context does not contain sufficient information to answer, "
                "say so explicitly rather than guessing."
            ),
        },
        {
            "role": "user",
            "content": f"Context:\n\n{context_text}\n\nQuestion: {query}",
        },
    ]

    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        temperature=0.1,
        max_tokens=1024,
    )
    return response.choices[0].message.content

Low temperature (0.1–0.2) keeps generation close to the retrieved context and reduces creative interpolation. The explicit instruction to acknowledge insufficient context prevents high-confidence hallucination, one of the most damaging production failure modes. We cover guardrails, hallucination detection, and faithfulness scoring in depth in Part 3: Monitoring and Continuous Improvement.

End-to-End Development Architecture#

Here is the complete development-phase architecture with all stages connected.

Summary table:

Common Mistakes in Development#

Skipping metadata on day one. Every developer knows they should add metadata "later." Later is always after a production incident. Add source path, document ID, chunk index, content hash, and creation timestamp before the first document is ingested.

Using one chunk size for all document types. A 512-character chunk works well for technical documentation and poorly for legal contracts with 200-word sentences. Use separate ingestion pipelines for meaningfully different document types.

No content hash or deduplication. The same document ingested twice doubles your index size and injects duplicate results. Hash every chunk at ingestion time and check before inserting.

Dense-only retrieval in production. Hybrid search (dense + BM25) almost always outperforms dense-only search for real-world query distributions that include specific terms, identifiers, and named entities. The implementation cost is low; skip it only if you have measured no improvement on your data.

No reranking step. Skipping reranking to save latency is a common shortcut. A cross-encoder rerank of 20 candidates to 5 adds 100–300ms but often improves answer quality by more than any other single pipeline change.

Not logging retrieval. If you cannot inspect which chunks were retrieved for a failed query, you cannot debug it. Log chunk IDs and scores from day one. This is the data that Part 3 of this series depends on.

Using the same embedding model for ingestion and query but updating it mid-stream. When you upgrade an embedding model, all existing vectors become incompatible. Schedule full re-ingestion before the upgrade goes live. Never mix vectors from two different model versions in the same index.

Building evaluation after production. Build a labeled test set of representative queries and expected chunks before you go live. It is the only way to validate that pipeline changes improve rather than regress retrieval quality. See Build a RAG Search Pipeline for an evaluation loop implementation.

Key Takeaways#

Building a RAG pipeline that works in production requires getting every layer right before the first document is ingested. Retrofitting metadata schemas, chunk strategies, and index types into a live system is painful and risky.

The architecture decisions with the most leverage are: metadata schema (determines what you can filter and delete), chunking strategy (determines retrieval precision), embedding model (determines semantic coverage), and whether you include a reranking step (determines final context quality). Dense-only retrieval without reranking is the most common reason a RAG demo outperforms the production system.

Every stage is measurable. Log retrieval results, scores, and LLM outputs from the first day. Without that data, debugging is guesswork. Parts 2 and 3 of this series build on those logs to manage deployment and drive continuous improvement.

The fundamental discipline of production RAG is not AI. It is data engineering. Clean ingestion, correct schemas, ACID-safe writes, versioned documents, and structured observability are what separate a demo from a system.

FAQ#

How do I choose between pgvector and a dedicated vector database?

Start with pgvector if you are already on PostgreSQL and your index will stay under 10 million vectors. You avoid a new infrastructure dependency and get SQL metadata filtering for free. Move to a dedicated vector database when you need native hybrid search, advanced payload filtering, or scale beyond what pgvector can serve at your latency target. See Pinecone vs Weaviate vs Qdrant for a detailed decision framework.

What chunk size should I start with?

512 characters with 64-character overlap is a reasonable starting point for technical documentation. Move to 768–1024 for documents with long sentences (legal, medical). Always benchmark against your actual query distribution. Measure retrieval hit rate at multiple chunk sizes before deciding. Intuition is unreliable.

Do I always need a reranking step?

Not always, but usually yes. Reranking is one of the cheapest quality improvements relative to its impact. Skip it only if your latency budget is extremely tight and you have measured that retrieval precision is already high enough without it. When in doubt, add reranking.

Should I use hybrid search or dense-only search?

Use hybrid search for any production application. Dense search misses exact keyword matches and specific identifiers. BM25 misses semantic paraphrases. Reciprocal Rank Fusion combines both with a single, parameter-free formula. The implementation cost is low and the recall improvement on real queries is consistent.

How do I handle document updates without re-embedding everything?

Use a content hash per chunk. At ingestion time, compute the SHA-256 of each chunk's text. If the hash matches the stored hash, skip embedding. If it differs, delete the old chunk row and insert the new one. This means you only pay embedding costs for content that actually changed.

What is the right temperature for RAG generation?

Use 0.1–0.2 for factual question-answering where the answer should be grounded in context. Higher temperatures allow the model to "fill in" missing context creatively, which feels helpful but increases the risk of confident hallucination. For RAG, lower temperature is almost always correct.

Where do hallucinations come from in RAG systems?

Hallucinations in RAG come from three places: retrieval failure (the right context was never retrieved), prompt failure (the model was not instructed clearly to stay grounded), and model tendency (some models hallucinate more readily than others). Parts 2 and 3 of this series cover detection and mitigation in depth.