At some point in building an AI feature, you change the prompt, the retrieval logic, or the model, and you have no reliable way to know whether the outputs got better or worse. So you merge anyway, or you spend an hour clicking through examples by hand. Neither is sustainable.
The fix is an evaluation pipeline: a structured way to run your AI feature against a representative set of inputs, grade each output, and compare the score to the previous baseline. When a change degrades quality, the eval catches it before the change ships.
This guide walks through building that pipeline from scratch. By the end you will have a working eval loop, a grading setup, and a clear picture of how to track regressions across model and prompt changes.
TLDR
- An eval pipeline is four parts: a dataset of representative inputs, a grader that scores outputs, a runner that executes the feature, and a baseline comparison.
- Start with cheap heuristic graders, then add an LLM-as-a-judge for the quality dimensions heuristics miss.
- You do not need a platform to start. A Python script and a JSON file will do until the pain demands more.
What you will build#
By the end of this guide, you will have:
- A dataset of representative input/expected-output pairs.
- A grader function that scores each output (heuristic and LLM-as-a-judge).
- A runner script that executes the AI feature against the dataset and records results.
- A baseline comparison that flags regressions.
The example uses a question-answering feature over a document corpus, but the pattern applies to any AI feature: summarization, classification, code generation, or agent workflows.
Prerequisites#
- A working AI feature (even one that is still rough is fine).
- Python 3.9 or later.
- An OpenAI API key, or a local model API that accepts OpenAI-compatible requests.
- Optional: LangSmith or Langfuse if you want a managed eval runner and trace storage.
Setup overview#
The eval pipeline has four components that flow into each other. The dataset feeds the runner, the runner calls your feature and hands each output to the grader, and the grader scores roll up into a report that flags regressions.
You can build all four in one script to start. As the dataset grows and evals get slower, the runner can be parallelized and the report can be stored to a database.
Step-by-step implementation#
Collect Representative Examples
The dataset is the most important part of the eval. A small dataset of well-chosen examples is more useful than a large dataset of easy cases.
Aim for 20–50 examples to start. Include:
- Typical queries that represent the most common use cases.
- Edge cases that have caused problems in manual testing.
- Adversarial inputs that try to break the feature (empty input, very long input, off-topic queries).
- Regression examples from real failures, meaning anything a user complained about.
Format each example as a JSON object with at minimum an input and an expected field:
[
{
"id": "q001",
"input": "What is the refund policy for annual subscriptions?",
"expected": "Annual subscriptions can be refunded within 30 days of purchase for a full refund.",
"tags": ["billing", "policy"]
},
{
"id": "q002",
"input": "How do I export my data?",
"expected": "Data export is available from Settings > Data > Export as CSV or JSON.",
"tags": ["data", "settings"]
}
]
The expected field does not need to be the exact output. It can be a reference answer that the grader uses as a baseline for comparison. For subjective tasks like summarization, the expected field can be a description of what a good output looks like rather than a literal string.
Save the dataset to evals/dataset.json. Keep it in version control, because it is as important as your code.
Write a Heuristic Grader
Start with heuristics before reaching for an LLM judge. Heuristics are fast, free, and deterministic. They catch obvious failures without needing a model call.
Common heuristic checks:
def heuristic_grade(output: str, expected: str, input: str) -> dict:
scores = {}
# Non-empty output
scores["non_empty"] = 1 if output.strip() else 0
# Length sanity (not too short, not absurdly long)
scores["length_ok"] = 1 if 20 <= len(output.split()) <= 500 else 0
# Key term presence (simple keyword match against expected)
expected_terms = [w.lower() for w in expected.split() if len(w) > 4]
output_lower = output.lower()
matched = sum(1 for term in expected_terms if term in output_lower)
scores["keyword_overlap"] = round(matched / max(len(expected_terms), 1), 2)
# No hallucinated disclaimer phrases
refusal_phrases = ["i don't have access", "i cannot answer", "as an ai language model"]
scores["no_refusal"] = 0 if any(p in output.lower() for p in refusal_phrases) else 1
scores["overall_heuristic"] = round(sum(scores.values()) / len(scores), 2)
return scores
Heuristics will not catch everything, but they catch the worst failures fast and cheaply. A score of 0 on non_empty or no_refusal is always a real failure that the LLM judge would also catch.
Add an LLM-as-a-Judge Grader
Heuristics fall short for quality judgments that require semantic understanding: is this answer factually correct, is it relevant, is it complete. An LLM judge can evaluate these dimensions at scale.
OpenAI's evaluation guide describes evals as structured tests that measure whether model outputs meet style and content criteria. LLM-as-a-judge is one of the recommended grading patterns, where a second LLM evaluates the primary model's output using a rubric.
A minimal LLM judge implementation:
from openai import OpenAI
client = OpenAI()
JUDGE_PROMPT = """You are evaluating the quality of an AI assistant's answer.
Question: {input}
Reference answer: {expected}
AI answer: {output}
Score the AI answer on a scale of 1 to 5 for each criterion:
- Factual accuracy: Does the answer match the reference on key facts?
- Completeness: Does the answer address the question fully?
- Relevance: Does the answer stay on topic?
Respond in JSON: {{"factual_accuracy": N, "completeness": N, "relevance": N, "reasoning": "..."}}
"""
def llm_grade(output: str, expected: str, input: str) -> dict:
prompt = JUDGE_PROMPT.format(input=input, expected=expected, output=output)
response = client.chat.completions.create(
model="gpt-4o-mini", # Use a cheaper model for judging
messages=[{"role": "user", "content": prompt}],
response_format={"type": "json_object"},
)
result = json.loads(response.choices[0].message.content)
result["overall_llm"] = round(
(result["factual_accuracy"] + result["completeness"] + result["relevance"]) / 15, 2
)
return result
Use a smaller, cheaper model for judging when possible. gpt-4o-mini is typically sufficient for rubric-based scoring and dramatically reduces cost at scale. Reserve the larger model for edge cases where the judge itself seems uncertain.
Chain-of-thought improves judge reliability
Ask the judge to reason before scoring. The reasoning field in the JSON response forces the model to think through its judgment before committing to a score, which reduces arbitrary scoring. Review the reasoning field during calibration to ensure the judge is applying your rubric correctly.
Write the Runner
The runner loads the dataset, calls your AI feature per example, applies both graders, and writes results to a file.
import json
import datetime
from pathlib import Path
def run_eval(dataset_path: str, output_path: str):
with open(dataset_path) as f:
dataset = json.load(f)
results = []
for example in dataset:
# Call your actual AI feature
output = your_ai_feature(example["input"])
heuristic_scores = heuristic_grade(output, example["expected"], example["input"])
llm_scores = llm_grade(output, example["expected"], example["input"])
results.append({
"id": example["id"],
"input": example["input"],
"expected": example["expected"],
"output": output,
"heuristic": heuristic_scores,
"llm_judge": llm_scores,
"timestamp": datetime.datetime.utcnow().isoformat(),
})
# Aggregate
avg_heuristic = sum(r["heuristic"]["overall_heuristic"] for r in results) / len(results)
avg_llm = sum(r["llm_judge"]["overall_llm"] for r in results) / len(results)
report = {
"run_id": datetime.datetime.utcnow().strftime("%Y%m%d_%H%M%S"),
"dataset_size": len(results),
"avg_heuristic_score": round(avg_heuristic, 3),
"avg_llm_score": round(avg_llm, 3),
"results": results,
}
with open(output_path, "w") as f:
json.dump(report, f, indent=2)
print(f"Eval complete. Heuristic: {avg_heuristic:.3f}, LLM: {avg_llm:.3f}")
return report
Run the eval before making changes to establish a baseline, then again after each significant change. Save both outputs to evals/runs/ with the run ID as the filename.
Compare to Baseline
The regression check is what makes the eval pipeline useful for shipping decisions. Without a comparison, you have scores but no judgment.
def compare_to_baseline(current_path: str, baseline_path: str, threshold: float = 0.05):
with open(current_path) as f:
current = json.load(f)
with open(baseline_path) as f:
baseline = json.load(f)
heuristic_delta = current["avg_heuristic_score"] - baseline["avg_heuristic_score"]
llm_delta = current["avg_llm_score"] - baseline["avg_llm_score"]
print(f"Heuristic delta: {heuristic_delta:+.3f}")
print(f"LLM judge delta: {llm_delta:+.3f}")
if heuristic_delta < -threshold or llm_delta < -threshold:
print("REGRESSION DETECTED — review individual failures before shipping.")
return False
print("No significant regression detected.")
return True
A threshold of 0.05 (5 percentage points on a 0–1 scale) is a reasonable starting point. Adjust it based on how much variance you observe across baseline runs of the same feature without any code changes. That natural variance sets your noise floor.
Verify the setup#
Once all four pieces exist, the day-to-day loop is short: run the eval to set a baseline, make a change, run again, and let the comparison decide whether the change is safe to ship.
Run the eval against the current feature to establish a baseline:
python eval_runner.py --dataset evals/dataset.json --output evals/runs/baseline.json
Then make a small intentional change (like shortening the system prompt) and run it again:
python eval_runner.py --dataset evals/dataset.json --output evals/runs/run_002.json
python compare.py --current evals/runs/run_002.json --baseline evals/runs/baseline.json
If the comparison correctly flags the intentional degradation, the pipeline is working. If it does not catch the change, revisit your grader rubric. It may not be sensitive to the dimension you changed.
Common mistakes#
Eval set only contains easy cases. If the dataset does not include edge cases and known failures, the eval will pass even when the feature degrades on the long tail. Deliberately add examples from past failures and support tickets.
Using the same model to judge and generate. If GPT-4o generates the answers and GPT-4o judges them, the judge may have a favorable bias toward its own outputs. Use a different model family, or use a smaller model as judge. It is less likely to be systematically biased toward the primary model's style.
Treating the eval score as a hard pass/fail. An eval score is a signal, not a gate. A small regression on one dimension might be acceptable if another dimension improved significantly. Review the per-example results on failing evals, because the aggregate score often hides the useful information.
Rebuilding the eval dataset from scratch after every refactor. The eval dataset is a long-lived artifact. Keep it in version control alongside the code. Add examples over time; do not replace them wholesale unless the feature scope has fundamentally changed.
Production considerations#
Run evals in CI. Integrate the eval runner into your CI pipeline to catch regressions before merging. Keep the dataset small enough that the CI eval completes in under a few minutes. A 30-example dataset with a cheap judge model typically completes in under 60 seconds.
Track scores over time. Store each eval run result in a database or S3 bucket. Plot the aggregate score across runs over time. Sudden drops correlate with specific commits and make root-cause analysis faster.
Managed eval platforms. LangSmith and Langfuse both support dataset management, eval runners, and score tracking in a UI. These are worth adopting when the manual runner becomes difficult to maintain or when you need to share eval results across a team.
Online evaluation complements offline eval. The pipeline built here is offline evaluation: it runs against a fixed dataset before shipping. Online evaluation scores a sample of real production traffic continuously, which catches failures that the eval dataset did not anticipate. See observability for AI systems for how to set up the production monitoring layer.
For teams extending into RAG features, the eval patterns here apply directly. See evaluating RAG quality for the retrieval-specific dimensions and metrics.
Key Takeaways#
An AI eval pipeline does not have to be complex to be useful. A 30-example dataset, a heuristic grader, an LLM judge, and a baseline comparison is enough to catch the majority of regressions before they reach users.
The discipline matters more than the tooling. An eval that runs before every significant change creates a forcing function: you cannot ship a prompt or model change without knowing whether it degraded anything measurable.
Start with the dataset. Build the grader. Run the baseline. Then add examples from every failure the feature produces in the real world. The eval set grows with the feature's history, and that history is what makes the eval useful.
FAQ#
How many examples do I need in an eval dataset?
Start with 20–50. That is enough to catch systematic regressions without making the eval slow or expensive to run. Add examples continuously from real production failures and edge cases. A mature AI feature might have 200–500 examples covering a wide range of input types. Prioritize diversity over volume.
What is LLM-as-a-judge and when should I use it?
LLM-as-a-judge uses a second language model to evaluate the primary model's output against a rubric. Use it when heuristics cannot capture the quality dimension you care about, such as factual accuracy, semantic relevance, or tone. It is not free: each judge call costs tokens. Run it on every example in CI and on a sample in production monitoring.
How do I handle non-deterministic outputs in my eval?
Run each example multiple times (3–5 runs) and average the scores to smooth out variance. If variance is high, where the same input produces wildly different scores across runs, that itself is a signal that the feature is unstable. Consider adding consistency as an explicit grading dimension.
Can I use my eval to compare two different models?
Yes. Run the full dataset through both models, grade each output set independently, and compare the aggregate scores. This is the basic form of A/B model evaluation. The same approach works for comparing prompt variants, retrieval configurations, or chunking strategies. OpenAI's evaluation best practices guide covers pairwise comparison approaches for model selection.
What is the difference between offline and online evaluation?
Offline evaluation runs a fixed dataset against your feature before shipping. Online evaluation scores real production traffic continuously. Both are needed: offline catches regressions before they reach users, online catches failures that the eval dataset did not anticipate. Start with offline, then add online monitoring once the feature is in production.
