AI evaluation is how a team decides, with evidence, whether an AI feature behaves well enough to ship. When the feature runs on a large language model, it’s often called LLM evaluation, or evals. It replaces “the demo looked fine” with written criteria, a fixed set of test inputs and a recorded decision.
Every team that adds a chatbot, a summarizer or an agent to its product faces the same question: how do we know it works? Conventional tests answer that for deterministic code. They struggle with non-deterministic features, where the same input can produce a different answer each time.
This article explains what AI evaluation is, why it differs from conventional testing, and how its parts fit together. It takes a software-testing point of view: requirements, test cases, verdicts and release decisions.
A working definition of AI evaluation
AI evaluation measures an AI feature’s outputs against explicit criteria, on a representative set of inputs, again and again over time. Its result is a verdict someone can act on, not just a score.
Three parts of that definition do most of the work:
-
Explicit criteria. “Good” is written down: accurate, grounded in the right source, polite, within policy, fast enough.
-
Representative inputs. The test set resembles what real users ask, including the awkward cases.
-
Again and again. Every change to the model, prompt, knowledge base or code is evaluated again, the same way.
Two kinds of evaluation share the name. Model benchmarks compare foundation models on general tasks such as reasoning or coding. Feature evaluation, the subject here, asks whether your product does its specific job for your users.
The difference matters in practice. A model can top a public leaderboard and still give your customers the wrong refund policy.
Why AI features break the usual testing model
Many automated tests work because behavior is deterministic: given a known state and input, a test can assert what should happen. AI features weaken that assumption in five ways.
-
The output isn’t fixed. The same question can get different answers on different runs. One passing run says little about the next.
-
There is rarely one right answer. “Our refund window is 30 days” and “You have 30 days to ask for a refund” are both correct. Exact matching fails good answers.
-
Quality has several dimensions. An answer can be accurate but rude, polite but invented, or correct while leaking a customer’s email address. A single pass or fail can’t hold all of that.
-
Failures look plausible. A wrong answer is usually fluent and confident. Nothing crashes, so nothing raises an alert.
-
Behavior changes without a code change. A new model version, a reworded prompt or a re-indexed knowledge base can shift answers. None of them shows up in a code review.
None of this makes testing impossible. It means a test has to describe quality instead of an exact value, and has to look at more than one answer.
The building blocks of an AI evaluation
Most AI evaluations, whatever the tooling, are built from the same five parts.
-
A dataset of test inputs. Often called a golden set: the questions or tasks the feature must handle. A good one mixes typical cases, edge cases and deliberately difficult ones. Each input has a stable ID and, where useful, a reference answer and a risk level.
-
Criteria, organized in a rubric. Each criterion names one quality dimension and a scale: pass or fail, named levels such as Poor to Excellent, or a number such as 1 to 5. Anchors describe what each level looks like, so two scorers reach the same result.
-
Scorers. Someone or something applies the rubric: people, code checks, or another model acting as a judge.
-
Repeated samples. Each input runs several times, because one answer is not enough evidence. Three to five runs is a sensible start; catching rare or high-risk failures can take many more.
-
Thresholds and rules. A pass rule for each criterion, rules for how results combine, and hard rules for failures that must never happen.
Two more things keep those parts trustworthy: versioning and a record of what was tested. A dataset or rubric that changes silently makes old results meaningless. So does a missing record of which model and prompt produced them.
Who scores AI outputs: code checks, people and LLM judges
There are three kinds of scorer, and mature teams use all three for different criteria.
| Scorer | Strengths | Limitations |
|---|---|---|
| Code checks | Format, length, required phrases, latency, cost, forbidden patterns such as card numbers | Meaning, tone and correctness of open-ended answers |
| People | Judgment, domain knowledge, subtle errors, tone | Speed, cost and consistency between scorers without clear anchors |
| LLM judges | Scale and speed on criteria that need reading comprehension | Systematic biases, so they need checking against people |
Use code checks wherever a rule can be written down. “Responds in under two seconds” and “never shows a card number” are rules, not opinions.
People are usually the reference for criteria that need judgment, especially where domain knowledge or risk is involved. Clear anchors make their scores consistent. Scoring only a sample of outputs keeps the effort manageable.
An LLM judge (often called LLM-as-a-judge) is another model, prompted with the rubric, that scores outputs. It makes evaluation scale, but judges can have systematic biases and can disagree with human reviewers. Treat a judge as an instrument to calibrate: have people score a sample without seeing the judge’s score, compare, and rely on the judge only for criteria where they agree.
Learn more: AI Testing Evaluators for Scalable, Reliable QA.
Repeated samples: consistency is a quality of its own
A feature that answers correctly only two times out of three isn’t reliably working, however good one demo looked. Repeated samples are how evaluation measures that.
Running each input several times exposes three things a single run hides:
-
Instability: the same question gets a right answer, then a wrong one.
-
Rare failures: a harmful or leaking answer that shows up in one sample out of many.
-
Spread: scores that vary widely even when the average looks fine.
How results combine matters as much as running them. Two excellent answers and one invented policy can average out to “good”. Better rules also look at the failure rate across samples and the spread of scores. For high-risk criteria, one failing sample is enough to fail the input.
From scores to a release decision
Scores are evidence; a verdict is a conclusion. Evaluation turns many small scores into one answer a team can act on, one step at a time. The four steps are: score each sample, judge each input, give the run a quality verdict and make the release decision.
Each step has its own rule. A sample passes or fails each criterion against its pass rule. An input passes when its samples do, with a review band for borderline results.
The run’s quality verdict (Pass, Needs review or Fail) follows from how many inputs pass. Hard rules skip the arithmetic: one leaked email address fails the run, whatever the average says.
The release decision is a separate step, and a person owns it. “Is it good enough?” and “Can we ship it?” are different questions. When a team ships despite a failing verdict, it records the reason, the person and the date.
Kinds of AI evaluation across the lifecycle
Evaluation isn’t one event before launch. Different kinds answer different questions at different moments.
| Kind | When | Question it answers |
|---|---|---|
| Development runs | While building or tuning | Is this prompt or model better than the last one? |
| Regression checks | On every change to code, prompt, model or data, often in CI | Did this change make anything worse? |
| Release evaluation | Before shipping | Is it good enough to release, and who agrees? |
| Red teaming | Before release, then periodically | Can deliberate attacks make it leak data, ignore its instructions or act outside its role? |
| Production monitoring | After release | Is it still behaving well on real traffic? |
Keep experiments and official runs apart. A team tuning a prompt may run dozens of experiments a day; they answer “did this change help?”, not “can we release?”. Only deliberate, reproducible runs, usually from the main branch or a release candidate, should enter the release record.
Production also feeds back into testing. A failure found in a real conversation is the most valuable test case a team can add, as long as the bad answer never becomes the expected one.
Making evaluation results comparable
Most evaluation is comparison: this version against the last, this model against that one. A comparison only means something when everything except the change under test stays fixed.
-
Pin versions. Record the dataset, rubric and threshold versions each run used. Compare runs only on the same versions, or only on inputs that didn’t change.
-
Record the configuration. Model and version, prompt version, knowledge base, retrieval settings and build. Without them, “what changed?” has no answer.
-
Change one thing at a time where you can. Swapping the model and the prompt together tells you that something changed, not what.
-
Respect noise. With repeated samples, small differences between runs can be chance. A one-point dip in an average may be noise; a newly failing critical input is not.
-
Read the answers, not only the numbers. The fastest way to understand a regression is to put the old and new answers to the same input side by side.
A worked example: a refund-policy assistant
A support assistant answers refund questions. The requirement says it must state the 30-day refund window, offer store credit after that, and never expose payment details.
The team builds its evaluation from the five parts:
-
Dataset: 120 questions, a mix of real and team-written, including edge cases such as gifts, sale items and closed bank accounts. Questions that touch payment details are marked critical.
-
Rubric: Accuracy (1 to 5, passes at 4), Groundedness in the policy documents (1 to 5, passes at 4), Tone (Poor to Excellent, passes at Good), and two pass-or-fail criteria: Follows refund policy, and No personal data leaked, where any failure fails the whole run.
-
Code check: response time under two seconds.
-
Samples: three answers per question.
A new prompt version arrives. The run shows 104 of 120 inputs passing, down from 110 on the previous version. The scores say “slightly worse”.
The answers say something more specific. Asked “Can I get a refund after 45 days?“, the assistant answers correctly twice. The third time it invents a 60-day window and claims to have started a refund.
That changes the decision. “Slightly worse” might ship; “sometimes invents policy” shouldn’t. The team holds the release, traces the problem to a retrieval change, and promotes that question into a smaller regression set that runs on every future change.
Common AI evaluation pitfalls
-
Testing once. One good run is a demo, not evidence. Run each input several times.
-
Averaging away failures. A high average can hide a critical failure. Keep hard rules for what must never happen.
-
Vague criteria. “Helpful” without anchors produces scores that disagree with each other. Describe what each level looks like.
-
Trusting an unchecked judge. An LLM judge nobody has compared with people is an opinion with a number attached.
-
A dataset that never grows. A fixed test set misses the failures users actually hit. Add every real failure as a new case.
-
Only synthetic inputs. Generated questions test what the team imagined. Mix in real ones as soon as you can.
-
Mixing experiments with official results. Dozens of trial runs a day make dashboards noisy and can change a requirement’s status by accident.
-
Keeping sensitive data you don’t need. Evaluation records can hold customer conversations. Decide what to store, redact it before storing, and limit who can see it.
-
Turning bad output into test truth. A model’s answer must never become a reference answer until a person has checked it.
Getting started with AI evaluation
A first evaluation needs no special platform. A spreadsheet and a short script are enough.
-
Pick one feature and one requirement. Start where a wrong answer costs the most.
-
Write three to five criteria with anchors. Include at least one hard rule for what must never happen.
-
Collect 15 to 30 inputs to start. Use real questions where you can, plus a handful of edge cases. Grow the set as real failures appear.
-
Run each input three times, and score the first round by hand. You’ll learn more about your criteria this way than from any tool.
-
Set pass rules for each criterion and for the whole run, before you look at the next results.
-
Record what you tested: model, prompt, knowledge base and dataset version.
-
Repeat on every change. Compare with the previous run, and read the answers that changed.
-
Keep every failure as a new test case, with a reference answer written by a person.
Automate once this works by hand. Add code checks for what can be measured, an LLM judge calibrated against your own scores, and checks in CI so a regression stops a merge.
AI evaluation is testing, adapted
AI evaluation is testing, adapted to software whose output isn’t fixed. Its core ideas will be familiar to any QA team: requirements, test cases, pass rules, regressions and a release decision someone owns.
What changes is the evidence. A rubric replaces the single expected value. Repeated samples replace the assumption that one output represents the system. And a decision that weighs how often and how badly the feature fails replaces a green build.
Teams that work this way ship AI features they can explain: what was tested, how it performed, and who decided it was good enough.
