Evaluation methods for AI features

A golden dataset is a collection of real production inputs paired with documented quality criteria — not exact expected outputs, but clear specifications of what a good output looks like for each input. It is the most valuable thing a QA engineer can build for an AI product, and the most underinvested. Teams that skip it have no reliable way to know whether a prompt change or model upgrade has regressed quality, and end up relying on vibes and the occasional user complaint.

The dataset should be built from real user traffic, not synthetic inputs. Synthetic inputs miss the distribution of edge cases that real users generate — the ambiguous queries, the misspelled inputs, the out-of-scope requests. Start by sampling from production logs, stratified across the major user intent categories. Aim for coverage of edge cases and failure-prone inputs, not just the happy path. A dataset of 200 carefully chosen real examples will outperform a dataset of 2,000 synthetic ones.

Version the dataset like production code. Every change to the dataset — adding examples, removing stale ones, updating quality criteria — should go through a review process. Diff it, review it, tag releases. When you run an eval and the score drops, you need to be able to tell whether the model regressed or the dataset changed. Without versioning you cannot answer that question reliably.

Run automated eval against the dataset on every model update, prompt change, or retrieval configuration change. This is the equivalent of a CI test suite for your AI product. A score drop that blocks a deployment is not a failure of the eval system — it is the eval system doing exactly what it should. Set thresholds deliberately, not aspirationally: the threshold should reflect the quality bar your users actually experience, not the quality bar you wish they experienced.

LLM-as-judge is the technique of using one language model to evaluate the output of another. It is the only practical path to automated evaluation of qualities that require reading comprehension — factual accuracy, helpfulness, appropriate tone, groundedness in retrieved context. Rule-based filters catch obvious failures; LLM-as-judge catches the subtle ones. The trade-off is that a poorly designed judge is not just inaccurate — it is inaccurate at scale and at speed, which means you can ship regressions confidently.

Rubric design is where the work is. A vague rubric ("is the response helpful?") produces inconsistent judgements. A specific rubric ("does the response address all sub-questions in the user query, without introducing claims not present in the retrieved context, in three paragraphs or fewer?") produces judgements you can argue about with a product manager and calibrate against human raters. Invest the time in rubric design before the judge. A good judge of a bad rubric is still a bad eval.

Calibrate the judge against human raters before you trust it. Run the judge and human raters over the same sample of outputs, measure agreement, and look for systematic disagreements. Common patterns: judges that score too leniently on verbosity, judges that penalise stylistic variation the product team actually wants, judges that fail on domain-specific terminology they were not calibrated for. The calibration gap tells you where to improve the rubric or the judge prompt.

The canonical failure mode is a judge that is wrong about the same things the evaluated model is wrong about — typically because they share training data or architectural biases. If you are using the same model family to judge and to produce outputs, build in specific calibration checks for the failure modes most likely to be shared. Some teams use a different model family for judging specifically to avoid this. There is no perfect solution; there is only knowing what you are measuring and what you are not.

Automated evaluation cannot catch what it was not designed to catch. The rubric you write reflects the quality dimensions you already know matter. Human reviewers catch the ones you did not anticipate — the response that is technically accurate but condescending in tone, the summary that omits the one piece of information the user actually needed, the answer that passes every automated check and still feels wrong to a domain expert. Human-in-the-loop evaluation is not a fallback for when automation fails; it is the mechanism for discovering new things to automate.

Rating UI design has an outsized effect on data quality. A binary thumbs-up/thumbs-down interface is fast to complete and nearly useless for diagnosis — you know something is wrong but not why. A five-point scale with labeled anchors and a freetext field for the most common failure reason gives you actionable signal. The goal is the minimum cognitive load that still produces diagnostic data. Raters who are fatigued or confused produce noise, not signal.

Inter-rater agreement is the quality signal for your rating process itself. If two raters looking at the same output agree less than 70% of the time, the rubric is ambiguous, the rater population is too heterogeneous, or both. Low agreement is not a problem with the AI product — it is a problem with the evaluation design. Measure it explicitly, diagnose the disagreements, and update the rubric. Rater disagreement on a specific dimension is usually a sign that the dimension needs splitting into two more specific ones.

Sample strategically. Random sampling finds average-quality failures. Stratified sampling — oversampling edge cases, low-confidence outputs, recent traffic from new user segments — finds the tails. The most valuable human review time is spent on the outputs where automated evals are uncertain, not on outputs where they are confident. Route high-confidence automated failures to automated handling and low-confidence ones to human review, and your human eval budget produces more signal per hour.