RAG From Development to Post-Deployment Monitoring, Part 3: Monitoring and Continuous Improvement

You shipped the RAG pipeline. Retrieval latency looks fine. Error rates are zero. The team calls it done.

Three weeks later, a customer files a support ticket: the assistant confidently cited a policy that was updated months ago. Two weeks after that, answer quality quietly drops across an entire topic area because someone reorganized the knowledge base. Nobody saw either regression coming because nobody was looking.

A RAG pipeline degrades silently. The model still generates fluent text, the vector store still returns results, and HTTP 200 keeps arriving. Without a monitoring layer that watches retrieval quality, answer faithfulness, and content freshness, a regression is invisible until users report it.

RAG pipelines degrade without you noticing#

Most applications fail loudly. A database is down, an exception is thrown, a status code is 500. RAG failures are different: they are soft failures that pass every infrastructure health check.

The retriever returns five chunks. They just happen to be the wrong five chunks for today's query. The model generates an answer. It just happens to claim something the context does not support. The pipeline is "healthy" by every standard SRE metric while the answers are quietly wrong.

This is the fundamental gap that RAG monitoring fills. Parts 1 and 2 of this series covered building and deploying the pipeline with quality gates before shipping. This part covers what happens after the gates, when real users, real queries, and a changing corpus make your carefully validated setup drift from its deployment-day baseline.

Instrument every stage with tracing#

The first prerequisite for monitoring a RAG pipeline is structured tracing across every stage. Retrieval, reranking, prompt construction, and generation must each be a named, measurable span.

A flat request log tells you the total latency and final response. A structured trace tells you whether retrieval was slow, whether the reranker changed the ranking significantly, and whether the LLM call consumed more tokens than usual. These are different diagnostic signals and only the trace gives you causality.

Each stage becomes a span, and those spans feed the metrics, evaluation, and alerting layers that the rest of this guide builds on.

The standard for AI pipeline tracing is OpenTelemetry's GenAI semantic conventions, which define normalized attribute names for model calls: gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, and more. Using these standard attributes means your AI traces flow into the same observability backend as the rest of your infrastructure.

Tracing a RAG pipeline with OpenTelemetry#

Here is what a fully instrumented RAG request looks like in Python:

from opentelemetry import trace
from opentelemetry.trace import StatusCode

tracer = trace.get_tracer("rag-service", "1.0.0")

def handle_rag_query(user_query: str) -> dict:
    with tracer.start_as_current_span("rag_pipeline") as root:
        root.set_attribute("query.text", user_query[:500])   # truncate for safety
        root.set_attribute("query.length_chars", len(user_query))

        # Stage 1: Retrieval
        with tracer.start_as_current_span("retrieval") as ret_span:
            chunks = vector_store.search(user_query, top_k=20)
            ret_span.set_attribute("retrieval.top_k_requested", 20)
            ret_span.set_attribute("retrieval.chunks_returned", len(chunks))
            ret_span.set_attribute("retrieval.top_score", chunks[0].score if chunks else 0.0)
            ret_span.set_attribute("retrieval.min_score", chunks[-1].score if chunks else 0.0)

        # Stage 2: Reranking
        with tracer.start_as_current_span("rerank") as rerank_span:
            reranked = reranker.rerank(user_query, chunks, top_n=5)
            rerank_span.set_attribute("rerank.input_count", len(chunks))
            rerank_span.set_attribute("rerank.output_count", len(reranked))
            rerank_span.set_attribute("rerank.top_score", reranked[0].score if reranked else 0.0)

        # Stage 3: Generation
        with tracer.start_as_current_span("llm_generation") as llm_span:
            llm_span.set_attribute("gen_ai.request.model", "gpt-4o")
            llm_span.set_attribute("prompt.version", "rag-qa-v2.3")
            response = llm_client.chat(build_prompt(user_query, reranked))
            llm_span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens)
            llm_span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens)
            llm_span.set_attribute("gen_ai.response.finish_reasons", [response.choices[0].finish_reason])

        return {
            "answer": response.choices[0].message.content,
            "sources": [c.metadata for c in reranked],
            "trace_id": format(root.get_span_context().trace_id, "032x"),
        }

The trace_id on the response is important: it lets you link a user-reported bad answer back to the exact spans that produced it.

Purpose-built tracing tools#

General APM tools handle span data but lack AI-specific workflows like prompt version diffing and retrieval quality trending. Three tools are purpose-built for this:

• Langfuse: open-source, natively understands token usage, prompt versions, and LLM-as-a-judge scoring attached to traces. The data model maps traces to observations to generations, which matches the RAG pipeline shape naturally.
• LangSmith: tight integration with LangChain/LangGraph, strong dataset management for building eval test suites, and automated evaluation scoring on captured traces.
• Arize Phoenix: open-source, built on OpenTelemetry and OpenInference instrumentation, with built-in RAG-specific evaluators for retrieval relevance and hallucination detection. Runs locally or in cloud.

All three support linking evaluation scores back to individual traces, which is the foundation of the improvement loop later in this guide.

What to log per request#

Log every variable that could explain a quality difference between two requests, not just the final answer.

The goal is complete diagnostic replay: given a user complaint, you should be able to reconstruct exactly what retrieval returned, what the model received, and what it generated.

Score answer quality on live traffic#

You cannot catch quality regressions by watching latency and error rates alone. You need to measure the dimensions of RAG quality on a continuous sample of live traffic.

The four core dimensions, as defined by the RAGAS framework (arXiv:2309.15217), are:

These can be computed with RAGAS without human-labeled ground truth. RAGAS uses an LLM-as-judge to decompose claims and evaluate each dimension. Here is a minimal online evaluation loop:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

def run_online_eval_batch(sampled_traces: list[dict]) -> dict:
    """
    Run RAGAS evaluation on a sample of production traces.
    Called on a schedule (e.g., every hour on last N=200 traces).
    """
    data = {
        "question": [t["query"] for t in sampled_traces],
        "answer": [t["response"] for t in sampled_traces],
        "contexts": [t["chunk_texts"] for t in sampled_traces],   # list of lists
        "ground_truth": [""] * len(sampled_traces),               # not required for all metrics
    }
    dataset = Dataset.from_dict(data)
    results = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )
    return results  # dict of metric → mean score

Track these scores over time. A downward trend in faithfulness across the same prompt version is a retrieval or corpus problem. A drop in context recall after a reindex is a chunking or embedding issue.

LLM-as-judge and its caveats#

LLM-as-judge is the mechanism RAGAS and similar tools use to score dimensions like faithfulness without human annotation. A second LLM receives the question, retrieved context, and answer, then scores whether claims are grounded.

It is useful and scalable, but carry these caveats:

• Position bias: judges may favor responses that appear first or are longer, not necessarily more accurate (LLMs-as-Judges survey, arXiv:2412.05579).
• Self-enhancement bias: a judge from the same model family may inflate scores.
• Domain gaps: in specialized knowledge domains, LLM-judge agreement with human experts can fall to 60–68%.
• Non-determinism: the same trace can score differently across runs.

Mitigations: use a strong judge model (not the same one generating answers), calibrate with a hand-labeled sample monthly, and treat trends rather than individual scores as the signal. See our experiment in Evaluating RAG Quality for a deeper look at how these metrics behave.

Detect and mitigate hallucinations#

A faithfulness score below your threshold is not just a low number. It means the model generated a claim that cannot be traced to the retrieved context, which is a hallucination.

There are four practical layers for catching this:

1. Faithfulness scoring with RAGAS or similar. As described above: decompose claims, verify each against context, score the fraction that are grounded. Run on a continuous sample. Alert when the rolling mean drops below your baseline.

2. NLI-based entailment checks. Natural Language Inference models classify whether a hypothesis is entailed, contradicted, or neutral with respect to a premise. Applied to RAG: the context is the premise, an answer claim is the hypothesis. Models fine-tuned for this task (e.g., cross-encoder NLI models from the Sentence Transformers family) can run cheaply per-claim without an LLM call. Research on NLI for hallucination detection shows this correlates with Attributable to Identified Sources (AIS) scores and works as an economical baseline check.

3. Citation and attribution verification. If your pipeline includes inline citations (e.g., "[Source 3]"), verify programmatically that each cited source was actually in the retrieved set and that the cited text contains the claimed fact. This is deterministic (no LLM call needed) and catches a distinct class of fabrication.

4. Abstention and confidence thresholds. When retrieval returns no chunks above a minimum similarity score, the model should say "I don't have enough information" rather than generate from parametric memory. Build this gate explicitly:

MIN_RETRIEVAL_SCORE = 0.65
MAX_LOW_SCORE_CHUNKS = 2

def should_abstain(chunks: list) -> bool:
    """Return True if retrieval confidence is too low to answer reliably."""
    if not chunks:
        return True
    high_confidence = [c for c in chunks if c.score >= MIN_RETRIEVAL_SCORE]
    if len(high_confidence) < MAX_LOW_SCORE_CHUNKS:
        return True
    return False

# In the main handler:
if should_abstain(retrieved_chunks):
    return {
        "answer": "I don't have enough relevant information to answer this confidently.",
        "abstained": True,
        "retrieval_scores": [c.score for c in retrieved_chunks],
    }

Track abstained=True events in your traces. A spike in abstentions points to retrieval degradation or a category of queries your corpus does not cover.

Close the loop with user feedback#

The highest-quality signal for whether a RAG answer was useful comes from the user, not from any automated metric.

Two categories of feedback matter:

Explicit signals:

• Thumbs up / thumbs down on an answer
• User-submitted corrections ("the answer is actually…")
• Regeneration requests on the same query

Implicit signals:

• Copy or citation of the answer (positive)
• Follow-up query that reformulates the original question (likely dissatisfied)
• Session abandonment immediately after receiving an answer (possibly negative)
• Long pause before next action (indeterminate, but worth tracking)

Log all of these linked to the original trace ID. This creates a ground-truth quality signal that no automated scorer can replicate. A thumbs-down on a specific trace, linked to the exact retrieval set and prompt that produced it, is an immediately actionable failure case.

The most important downstream use of these signals: turn failures into eval cases. Every negative feedback event is a candidate for your golden evaluation set. The query, the bad answer, and what the correct answer should be form a regression test. Add it to your eval suite and run it before every future prompt or retrieval change. This is what production-driven improvement actually looks like.

Drift: the slow degradation you did not ship#

RAG pipelines can degrade without any code change. Drift is the accumulation of mismatches between the system as deployed and the world it operates in.

There are five drift types to track:

The continuous improvement loop#

Monitoring without acting on what you find is just expensive logging. The value is in the loop: detect, diagnose, improve, validate, ship.

Each production failure that you mine becomes a new eval case, which feeds the quality gate, which protects the next change. That feedback edge is what makes the loop compound over time.

Ship changes safely#

Never ship a retrieval or prompt change without first validating it against your offline eval set and then confirming it under A/B or shadow traffic.

The reason: a change that improves faithfulness on your golden set may degrade recall for a different query category. Offline eval catches known regressions; A/B or shadow tests catch unknown ones.

The workflow:

1. Change is proposed (retrieval config, prompt, embedding model)
2. Run offline eval → golden set pass rate must not regress
3. Deploy to shadow (receives production queries, responses not shown to users)
4. Compare shadow vs. production metrics for 24–72 hours
5. If no regressions: promote to canary (5–10% traffic)
6. Monitor canary metrics for latency, cost, faithfulness score
7. Full rollout or rollback

This connects directly to the deployment gates described in Part 2. The improvement loop feeds new eval cases into step 2, making the gate stronger with each cycle.

Monitor cost, performance, and alerts#

Every RAG request has a measurable cost. Without alerting, a slow prompt change or a chatty reranker can quietly multiply your bill.

What to track and alert on:

# Example alert rule (Prometheus/AlertManager style)
- alert: RAGFaithfulnessRegression
  expr: |
    avg_over_time(rag_faithfulness_score[1h]) < 0.70
  for: 30m
  labels:
    severity: warning
  annotations:
    summary: "RAG faithfulness score below threshold for 30 minutes"
    description: "Rolling 1h mean faithfulness: {{ $value }}. Check retrieval quality and recent prompt changes."

- alert: RAGHighAbstentionRate
  expr: |
    rate(rag_abstentions_total[30m]) / rate(rag_requests_total[30m]) > 0.10
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "RAG abstention rate above 10%"
    description: "Too many queries returning 'insufficient information'. Check corpus coverage and index health."

When an alert fires: first check whether a code or prompt change was recently deployed (roll back if confirmed). Then check the retrieval stage by looking at zero-chunk rate and average similarity scores. If retrieval looks healthy, check prompt version and model responses directly on recent failing traces.

Common monitoring mistakes#

Getting the instrumentation in place is only half the challenge. The other half is avoiding the patterns that make monitoring less useful than it could be.

Logging only the final response. The most common mistake. Without per-stage spans, you cannot distinguish a retrieval failure from a generation failure. Both look identical in flat logs.

Running evals only at deploy time. Offline eval before deployment is necessary but not sufficient. A corpus change, a user behavior shift, or an embedding model update can degrade quality without any deployment. Schedule eval runs on production samples continuously.

Treating every LLM-as-judge score as ground truth. Judge scores are trends, not verdicts. Validate a sample by hand monthly. If your judge consistently scores a known-bad answer highly, calibrate against your human labels.

Alerting on individual scores instead of trends. A single faithfulness score of 0.55 on one trace is noise. A rolling mean that dropped from 0.82 to 0.68 over three days is a signal. Alert on windows and trends, not point values.

Forgetting prompt version in the trace. Without prompt versioning, you cannot isolate whether a quality change was caused by a prompt update, a model update, or corpus drift. Tag every LLM call with the prompt template name and version hash.

Ignoring implicit user signals. Explicit feedback is rare. Most users do not click thumbs up or down. Implicit signals (reformulated follow-ups, abandoned sessions, copy events) are lower-signal individually but high-value in aggregate. Collect them.

Over-alerting on latency without context. A latency spike during a reranker upgrade or a large batch reindex is expected. Alert suppression tied to known maintenance windows prevents fatigue from false positives.

Key Takeaways#

• A RAG pipeline that looks healthy by standard infrastructure metrics may be silently producing wrong answers. HTTP 200 is not a quality signal.
• Trace every stage as a named span with attributes following OpenTelemetry GenAI semantic conventions. Purpose-built tools like Langfuse, LangSmith, and Arize Phoenix add AI-specific workflows on top.
• Log query, retrieved chunks with scores, reranked set, prompt version, model, response, latency, tokens, and cost per request. Full prompts should be sampled.
• Continuously evaluate faithfulness, answer relevance, context precision, and context recall using RAGAS (arXiv:2309.15217) on a sample of live traffic. Track score trends over time, not individual scores.
• Detect hallucinations at multiple layers: faithfulness scoring, NLI entailment checks, citation verification, and low-confidence abstention gates.
• Collect both explicit and implicit user feedback. Every negative feedback event is a regression test candidate.
• Monitor for five types of drift: data, embedding model, query distribution, retrieval index quality, and content freshness.
• The improvement loop is: mine failures → expand golden eval set → tune retrieval/prompt → validate offline → ship under a gate. Make this a repeating cycle, not a one-time sprint.
• Never ship a retrieval or prompt change without offline eval regression check and shadow/A/B confirmation first.

FAQ#

How often should I run online evaluations in production?

Run evaluations on a rolling sample continuously, not as a periodic batch job. A practical setup: evaluate a random 5–10% sample of requests in near-real time (minute-level delay is acceptable), and compute hourly and daily rolling averages for the quality metrics. This gives you enough temporal resolution to catch a regression the same day it is introduced without the cost of evaluating every request.

What is a good target faithfulness score for a production RAG system?

There is no universal benchmark, and you should calibrate against your own system baseline rather than an industry number. A practical approach: establish your baseline faithfulness score at launch on a representative sample, then set your alert threshold at 10–15% below that baseline. A freshly built RAG system on a clean, well-structured corpus will typically score above 0.75 with RAGAS faithfulness, but this varies considerably by domain, query type, and LLM.

Should I use RAGAS, a custom LLM judge, or NLI for hallucination detection?

Use them at different layers. RAGAS is best for continuous evaluation of answer quality across all four dimensions, and it is well-tested and reference-free. A custom LLM judge is best for domain-specific criteria that RAGAS does not cover (e.g., "does the answer follow our compliance phrasing rules"). NLI entailment checks are best for high-throughput per-claim grounding verification where cost matters, since they are cheaper than an LLM call and deterministic. Running all three on different samples or at different pipeline stages is not unusual for mature systems.

When should I re-embed the entire corpus?

Re-embed when you upgrade the embedding model, when you change tokenization or preprocessing that affects chunk boundaries, or when a significant fraction of your corpus has been replaced. Do not partial-update an index after an embedding model change, because mixing embedding generations makes cosine similarity meaningless. Also consider a full re-embed if your retrieval quality metrics show unexplained decay after a period of high write volume, which can degrade vector index graph quality.

How do I handle queries my RAG system should not answer?

Build an explicit abstention layer based on retrieval confidence, not just LLM refusals. If no retrieved chunk exceeds a minimum similarity threshold, return a structured "insufficient context" response rather than letting the model generate from parametric memory. Track these abstentions. A spike tells you either that users are asking about topics your corpus does not cover (a coverage gap to fix) or that retrieval quality dropped (an index or corpus freshness problem).

How does Part 3 connect back to Part 2's deployment gates?

Part 2 established offline eval as a merge gate: no retrieval or prompt change ships without passing the golden test set. Part 3's improvement loop feeds that gate directly. Every production failure you convert into a golden eval case makes the gate more comprehensive. Over time, the gate becomes a safety net that catches not just the regressions you expected but the ones you didn't. The monitoring layer in Part 3 is also what tells you which gate cases to add next.