You evaluate an LLM system you cannot see inside by testing its behaviour, not its internals: build a set of representative inputs with agreed expectations, score outputs against what actually matters for the task, run those checks continuously to catch regressions, and keep humans in the loop for the judgements that machines cannot make reliably. You treat it as an empirical, statistical problem rather than a deterministic one.
Why traditional testing does not apply
Conventional software testing rests on assumptions a language model breaks. Normal code is deterministic: the same input gives the same output, so you assert exact equality and move on. You can read the logic and reason about every branch. A language model offers none of that. The same prompt can produce different wording each time. There is no readable logic to inspect, only weights you cannot interpret. And correctness is rarely a single right string; it is a quality judgement across many acceptable answers.
So "assert output equals expected" fails immediately. A perfectly correct answer phrased differently from your fixture would fail the test, while a fluent, confident, wrong answer might pass a naive keyword check. Evaluating LLMs is closer to assessing an exam script than checking a return value. You need a way to judge quality across variation, and that changes everything about how you test.
Build an evaluation set
The foundation of everything is your evaluation set, often called an eval set: a collection of representative inputs paired with a definition of what a good response looks like for each. This is the single most valuable asset in a serious LLM project, and skipping it is why so many teams end up arguing about quality by vibes.
A strong eval set has a few properties:
- It is representative of real usage. Draw from actual or realistic inputs, not tidy examples the system obviously handles. If real users ask messy, ambiguous questions, your eval set must contain messy, ambiguous questions.
- It includes the hard cases on purpose. Edge cases, ambiguous requests, out-of-scope questions, and inputs that should be refused. These predict production behaviour far better than the happy path, and they are where systems actually break.
- It defines "good" per case. Depending on the task, that might be a reference answer, a set of facts the response must contain, a required format, or a rule such as "must decline and not invent an answer". The definition is the contract you score against.
- It grows from real failures. Every time production surprises you, that input becomes a permanent entry. Over time your eval set becomes a memory of every way the system has ever gone wrong, so it can never regress there silently.
Start smaller than you think, and prioritise honesty over size. A few dozen carefully chosen, genuinely representative cases beat hundreds of trivial ones.
Offline and online evaluation
There are two settings where you evaluate, and you need both.
Offline evaluation runs your system against the eval set before anything reaches users. It is your regression net and your development feedback loop: change a prompt, swap a model, adjust retrieval, then rerun the set and see whether scores moved up or down. Because it uses a fixed set with known expectations, it is repeatable and lets you compare changes fairly. This is what turns "I think this is better" into "this is measurably better on our cases".
Online evaluation measures what happens with real users in production. No eval set covers everything, and real usage always drifts from your assumptions. Online signals include explicit feedback such as thumbs up or down, and implicit signals such as whether users had to rephrase, whether they abandoned the task, or whether they escalated to a human. Offline tells you a change is safe to ship; online tells you whether it actually helped once it met reality. Trusting one without the other is how teams ship confident regressions.
Measure what matters, not model benchmarks
A common trap is evaluating against generic benchmarks or leaderboard scores. Those tell you how a model performs on someone else's academic tasks. They tell you almost nothing about whether it does your job well.
The only score that matters is task success on your use case. A model that tops every public benchmark can still be worse at your specific task than a humbler one. Measure the job, not the model's reputation.
Define success in terms a stakeholder would recognise. For a support assistant, did it resolve the issue correctly. For an extraction task, did it pull the right fields with the right values. For a system that must stay in bounds, did it correctly refuse what it should refuse. Then measure the failure modes that carry real cost, and weight them by consequence. A confidently fabricated answer is usually far worse than an honest "I do not know", so your scoring should punish the first much harder than the second. Evaluation is not one number; it is a small set of measures tied to what actually helps or harms the user.
Detect regressions automatically
LLM systems regress in ways normal software does not. A prompt tweak to fix one behaviour can quietly break three others. A model provider updates their model underneath you and yesterday's good outputs shift. Retrieval quality drifts as your data changes. None of this throws an error. It just gets worse.
The defence is to run your offline eval set automatically on every meaningful change, the same way you run a test suite, and to treat a drop in score as a failure that blocks release. Because scores are statistical rather than pass or fail, you compare aggregate results across cases rather than demanding every case pass. This mindset is entirely compatible with disciplined engineering practice; see our take on software development methodologies for how continuous checks fit a delivery process. Without an automated eval gate, you will ship regressions and hear about them from users, which is the most expensive possible way to find out.
Keep humans in the loop
Automation gets you far, but some judgements resist it. Whether a tone is right, whether a nuanced answer is genuinely correct, whether an output is subtly misleading: these often need human judgement. There are two ways humans stay in the loop, and mature systems use both.
- In evaluation. Periodically, real people review a sample of outputs and score them. This calibrates your automated measures, catches quality issues no automated check noticed, and keeps your definition of good honest as the product evolves.
- In production, for high-stakes cases. Where a wrong answer is costly, route uncertain or important outputs through a human before they take effect. The system does the heavy lifting and a person owns the final call, so the model's inevitable mistakes never land unchecked on the user.
Using a strong model as an automated judge of another model's output is a genuinely useful technique that scales evaluation, but it is a supplement to human judgement, not a replacement. Someone has to check that the judge itself agrees with people, or you have simply moved the trust problem rather than solved it.
The mindset shift
Evaluating an LLM system is not testing in the traditional sense. It is measurement of a probabilistic system through its behaviour: build a representative eval set, score against real task success, gate every change on it, watch production signals, and keep humans where judgement is required. Teams that treat evaluation as the core engineering discipline ship systems that improve deliberately. Teams that treat it as an afterthought ship systems that drift and surprise them.
If you are building an LLM system and want it evaluated with real rigour rather than hope, that is the work we do. See our approach to AI development or book a free consultation.