Most teams shipping an LLM feature test it the same way: someone types a few prompts, eyeballs the answers, says "looks good," and merges. Then a prompt tweak two weeks later quietly breaks a case nobody thought to retry, and the first person to notice is a customer.
The fix is an eval set — a fixed list of inputs with a way to score the outputs. It's the unit test suite of LLM development, except the assertions are fuzzier and the failures are more interesting. Teams put off building one because it sounds like a research project. It isn't. A genuinely useful first eval set is a one-day job, and this post is the plan for that day.
Why "looks good" stops working
Traditional code is deterministic: same input, same output, so one passing test stays passed. LLM features break that assumption twice. The model's output varies run to run, and — more importantly — every change you make is global. Editing a prompt to fix one bad answer changes the behavior on every input, not just the one you were staring at. Swapping models, adjusting temperature, reordering context: same story.
Without an eval set, every change is a trade you can't see. You verified the case in front of you and silently re-rolled the dice on everything else. This is why LLM development without evals feels like whack-a-mole — because it literally is: fixing one case regresses another, and you have no instrument that shows the regression.
An eval set turns that into a visible trade: "this prompt change fixed 4 of the 9 failing cases and broke 1 that used to pass." Now you're doing engineering again.
What a first eval set actually is
Strip away the tooling and an eval is three parts:
- Cases — a list of inputs, ideally with a reference answer or a note on what "good" means for each.
- A runner — a script that sends each input through your actual pipeline (same prompt template, same retrieval, same model settings as production) and collects outputs.
- A grader — something that scores each output: exact match, a code check, or another model acting as judge.
That's it. A JSONL file, a loop, and a scoring function. Resist the urge to start by evaluating vague qualities like "helpfulness" or "tone" — start with the failure modes that would actually embarrass you in production, because those are the ones you need to catch.
The one-day plan
Morning: collect 30–50 real cases
The single biggest quality lever is where your cases come from. In descending order of value:
- Real user inputs. If the feature is live (even internally), pull actual queries from your logs. Real users phrase things in ways you won't invent: half-formed questions, typos, two questions in one, questions your system can't answer.
- Bug reports and complaints. Every "the bot said something weird" message someone posted in Slack is a gold-plated eval case. It's a known failure — put it in the set so it can never silently return.
- Cases you write yourself. Fine for filling gaps, but be deliberate: for each important behavior, write the straightforward case and the adversarial sibling — the ambiguous phrasing, the question with a false premise, the input in the wrong language.
Aim for 30–50 cases. That's small enough to build in a morning and read in one sitting, and large enough that a score change means something. A hundred mediocre synthetic cases teach you less than thirty drawn from real traffic.
Make sure you include cases where the right behavior is to decline: questions outside the product's scope, requests for information that isn't in the documents, prompts trying to steer the bot off-task. Most homegrown eval sets only test "does it answer well" and never "does it refuse when it should" — and the second category is where the embarrassing screenshots come from.
Store cases in the most boring format available:
{"id": "refund-policy-basic", "input": "How do I get a refund?", "expected": "Mentions the 30-day window and links to the refunds page", "kind": "answer"}
{"id": "out-of-scope-medical", "input": "What dosage of ibuprofen should I take?", "expected": "Declines — out of scope, suggests consulting a professional", "kind": "refusal"}
A JSONL file in your repo, reviewed like code. No platform required.
Midday: write the runner
The runner's one job is fidelity: run each case through the same path production uses. If your feature does retrieval, run retrieval. If production sets temperature and a system prompt, use those. An eval that calls the model directly while production goes through three layers of prompt assembly is measuring a different system.
import json
def run_eval(cases_path, pipeline):
results = []
for line in open(cases_path):
case = json.loads(line)
output = pipeline(case["input"]) # your real production path
results.append({**case, "output": output})
return results
Save every run's raw outputs with a timestamp. When a score drops, the first thing you'll want is to diff the actual outputs between runs, not just stare at the number.
Afternoon: grade the outputs
Grading is where teams overthink it. Use the cheapest grader that catches the failure, and mix grader types across the set:
Code checks first. A surprising fraction of what you care about is mechanically checkable: Is the output valid JSON? Does it match the schema? Is it under the length limit? In the right language? Does the refusal case actually decline (no answer-shaped content)? Does the answer cite one of the retrieved documents? These graders are free, fast, and never disagree with themselves. Exhaust them before reaching for anything smarter.
String and pattern checks second. "Must mention the 30-day window" is a substring test. Crude, but crude-and-stable beats clever-and-flaky for a first set.
LLM-as-judge last, and narrowly. For genuinely fuzzy criteria — "is this answer supported by the provided context?" — use a second model call as the grader. Two rules make this work in practice:
- Ask a narrow, binary question per criterion. Not "rate this answer 1–10" (scores drift and mean nothing), but "Does the answer contain any claim not supported by the context? Answer yes or no, then quote the unsupported claim." Binary questions with required evidence are dramatically more consistent.
- Calibrate the judge before trusting it. Grade 15–20 outputs yourself, run the judge on the same ones, and compare. If you disagree with the judge often, tighten its instructions until you mostly agree. A judge you haven't checked against human judgment is just a second opinion of unknown quality.
JUDGE_PROMPT = """Context provided to the assistant:
{context}
Assistant's answer:
{output}
Does the answer contain any claim not supported by the context?
Reply with exactly "yes" or "no" on the first line.
If yes, quote the unsupported claim on the next line."""
By the end of the day you have: a case file, a runner, a mix of graders, and a baseline score. Total infrastructure: two short scripts.
Using it: the loop that makes it pay off
The eval set earns its keep through one habit: run it before and after every meaningful change. New prompt wording, a model upgrade, a chunking change in your RAG pipeline, a temperature tweak — run the set, diff the scores, read the cases that flipped.
Reading the flips is the important part. The aggregate score is a smoke alarm; the individual failures are the information. "Overall 84% → 86%" tells you little. "The model upgrade fixed all three multilingual cases but now answers the out-of-scope medical question" tells you exactly what to do next.
Two practices keep the set alive:
- Every production failure becomes a case. When something goes wrong in the wild, your first commit is the eval case that reproduces it — before the fix, so you can watch it flip from fail to pass. This is regression testing, and it works exactly as well for prompts as it does for code.
- Wire it into CI, with a floor rather than perfection. Nondeterminism means the score will wobble a little run to run; don't fail the build on a one-case dip. Fail it when the score drops below a floor you set, or when a case tagged must-pass (your known-embarrassing failures) breaks. Prompt changes now get the same protection as code changes.
What to skip on day one
Deliberately out of scope for your first set, and fine to skip:
- Eval platforms and dashboards. A JSONL file and a script are enough until the set has proven its value. Adopt tooling when you feel the pain it solves, not before.
- Statistical rigor. With 40 cases you can't detect a two-point quality difference, and you don't need to. You're catching regressions and comparing candidates, not publishing a benchmark.
- Evaluating everything. One feature, its top handful of failure modes. A narrow eval you run on every change beats a comprehensive one you run quarterly.
The pattern here mirrors testing generally: the first test suite is never complete, and that was never the point. The point is that from tomorrow onward, changes to your LLM feature get measured instead of eyeballed — and every new failure you encounter has a place to live where it can never surprise you twice.
Building an LLM feature and not sure how to tell if it's getting better or worse? Get in touch — we help teams design and build AI systems that fit the job.