That one question is the whole reason evals exist. An eval turns "it feels better" into "it scored better." Here is the idea in plain English, no code required.
Everyone is writing prompts. Almost nobody is checking whether the prompt worked. That check has a name: an eval. It is an exam you give the AI, and this whole page is that one idea, start to finish.
Forget the jargon. An eval is just an exam. A real exam has three parts you already know: the questions, the answer key, and the grade.
An eval has those same three, plus one more that matters most. You give the exam again, every time something changes. Hold that picture. We are not going to swap it for a different one.
Build them in order. Each part rests on the one before it. To keep it concrete, here is a real eval reviewers run on every new AI model, the senior engineer eval.
A set of real cases to test.
Here: a real production codebase that works but is a mess inside, the kind someone called "vibe-coded slop."What you ask the AI to do with each case.
Here: rewrite that codebase from first principles, the way a senior engineer would.Mark the answer against an answer key for what good looks like.
Here: two human senior engineers did the same rewrite first. Their work is the answer key, and every model is scored out of 100 against it.The part people miss. Give the same eval again every time you change the prompt or swap the model.
One grade is a snapshot. Re-running it is what turns a one-time check into an eval.Skip part four and you are back to a vibe check with extra steps. Keep it, and you have a number you can trust as everything else changes.
Here is the question every engineer asks, and it is the right one. The answer is the cleanest way to see what an eval really is.
A normal software test is deterministic. Same input, same output, every time. 2 + 2 is always 4, so you assert the exact answer and it passes or fails. That is the world QA and engineers already live in.
AI is non-deterministic. Ask it the same thing twice and you can get two different answers, both good. And there is rarely one correct answer to assert against in the first place. The moment you write "assert the output equals X," you have lost, because there are a hundred good versions of X.
So you stop checking for one exact answer and start grading quality on a scale. That switch, from a pass-or-fail assertion to a graded score, is the line between a test and an eval.
This is the whole point, so slow down here. Without an eval, you eyeball two or three answers and ship on a hunch. With one, you get a grade you can trust and repeat.
You eyeball a few answers and ship on a hunch. No record, no re-run, no idea whether the next change quietly made it worse.
You get a grade you can trust. Change something, take the exam again, and watch the number move.
Here is the senior engineer eval in full. The questions are a real production codebase, the kind that works but is a mess inside. The ask is blunt: rewrite it from first principles, the way a senior engineer would. To grade it, two human senior engineers did the exact same rewrite first, and their work becomes the answer key. Every model is scored out of 100 against what the humans built, which puts human level in the 80s and 90s. Then the same eval is re-run on each new model as it ships.
Build an eval hard enough to fail. If every answer passes, you learned nothing. The useful eval is the one some answers flunk, because the failures show you exactly where the AI breaks. For months, every model failed this one. That gap was the most honest thing on the page.
A higher score is not automatically good news. Up just means something changed. Maybe the AI got smarter. Maybe the eval was easy to game. Treat a jump as a reason to go look, not a reason to celebrate.
The concept outlasts the tools. Models change, prompts change, the tool you grade with this year is gone next year. The eval, your written idea of what good looks like, is the part you keep.
If you can't measure it, you're guessing. Evals are how you stop guessing.
Your move this week: write five test cases for the one AI task you rely on most. That is your first eval.