Evaluating a Prompt: The Four-Step Loop

Ch21 said the discipline begins with eval, because you cannot operate what you cannot measure. This chapter is the first and smallest unit of that measurement: a single prompt. The question “is this prompt good?” has a deceptively simple-looking answer — run it and see — but a one-shot look is exactly the vibe ch21 warned against. The thesis here is that knowing a prompt is good is a loop: you commit to what “good” means, build a way to measure it, iterate, and grade — and then go round again. The unit is a prompt; ch23 will take the same shape up to a whole agent trajectory.

The four-step loop

Anthropic frames building an LLM application as a cycle: first define success criteria, then build evaluations to measure against them — and “This cycle is central to prompt engineering.” [Official] Define success criteria and build evaluations · AnthropicT1-official original That single sentence is the spine of this chapter. Evaluating a prompt is not a checkpoint you pass once; it is a feedback loop you stay inside while the prompt is alive.

The loop has four steps, and they run in order:

1. Define success criteria — pin down what “good” means for this prompt, measurably.
2. Build test cases — assemble a representative set of inputs to run the prompt against, favoring automatable volume over a small hand-curated set (the opposite of ch23’s small, expensive trajectory suites).
3. Iterate with tooling — change the prompt (the prompt improver drafts a candidate) and re-run.
4. Grade outputs — score each run against the criteria, with a grader matched to the criteria.

Then you loop. The grade tells you whether the last change helped; if not, you iterate again. The shape is identical to how you treat code: a measurable target, a test set, an iteration tool, and a grader — looped until the bar is met. The rest of this chapter takes the four steps in turn, but the order itself carries the first lesson: steps 1 and 2 are not interchangeable with 3 and 4.

Criteria and tests are preconditions

The reason steps 1 and 2 come first is that Anthropic’s prompt-engineering overview lists them as prerequisites before prompt engineering begins. The first listed prerequisite is “A clear definition of the success criteria for your use case” [Official] Prompt engineering overview · AnthropicT1-official original , and the second is “Some ways to empirically test against those criteria.” [Official] Prompt engineering overview · AnthropicT1-official original (The third is a first-draft prompt to improve — the thing the loop then iterates.) The criteria and the test set are the entry gate to the loop, not artifacts you produce along the way.

And the criteria have to be measurable. The guidance is explicit: good criteria “Use quantitative metrics or well-defined qualitative scales.” [Official] Define success criteria and build evaluations · AnthropicT1-official original They are typically multidimensional, too — accuracy, output format, latency, and cost are different axes, and a prompt can win on one while losing on another. A criterion you cannot express as a number or a consistently applied scale is not a criterion; it is a hope.

This is the anti-vibes move, and it is the whole reason the loop exists. If you fix “what good is” and “how to measure it” before you start changing the prompt, then every later change is judged against a target that does not move. Improvement becomes something you measure, not something you assert. Invert the order — tweak the prompt first, then decide whether you like the output — and you have no fixed reference, so “it feels better now” is the best you can honestly say. It is the same attribute-first discipline good engineering applies everywhere: name the target, then chase it.

Engineering the eval set

The test set is the second precondition, and it is engineered, not merely collected. Two design principles do most of the work, and they pull against each other.

The first is fidelity: be task-specific. “Design evals that mirror your real-world task distribution” [Official] Define success criteria and build evaluations · AnthropicT1-official original — the set should look like the inputs the prompt will actually see in production, weighted the way they actually occur. And it must deliberately include edge cases: irrelevant or nonexistent input, overly long input, harmful input, ambiguous cases. A test set that only contains the happy path tells you nothing about the inputs that break things.

The second is throughput, and it is where most teams flinch: “More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.” [Official] Define success criteria and build evaluations · AnthropicT1-official original A large set you can grade automatically beats a small set you grade by hand — even though each automated grade is individually noisier. Volume buys statistical signal; hand-grading caps volume at whatever a human can sustain. So you structure the questions to be machine-gradable where possible (multiple-choice, string match, code-graded, or LLM-graded), and you prioritize covering the distribution over polishing a handful of items.

The tension between these two — real-world fidelity against automatable volume — is the central design problem of the eval set. You want it representative and large enough to grade cheaply at scale, and those goals trade against each other at the margin. Resolving that tension is exactly why the grading step matters so much, which is where the loop is heading.

Tool-assisted iteration

With the preconditions fixed, the loop’s third step is iteration — and Anthropic ships two complementary tools for it, one to draft the change and one to measure it.

The drafting tool is the prompt improver, which “helps you quickly iterate and improve your prompts through automated analysis and enhancement.” [Official] Console prompting tools · AnthropicT1-official original It proposes the next version of the prompt — it is reported to excel at making prompts more robust for complex, high-accuracy tasks, enhancing a prompt in steps (identifying examples, drafting, chain-of-thought refinement, example enhancement). Its companion on the same page, the prompt generator, drafts a first prompt from a task description. The improver is the generate half of iteration: it gives you a candidate to test.

The measuring tool is the Console Evaluation tool, which closes the loop on variants. Its prompt-versioning affordance lets you “Create new versions of your prompt and re-run the test suite to quickly iterate and improve results” [Official] Using the Evaluation Tool · AnthropicT1-official original , and it offers side-by-side comparison as the A/B mechanism — you put two or more prompt versions next to each other on the same test cases and read which one scores better. That is the measure half: the eval set plus versioning decides whether the candidate is actually an improvement.

So iteration is generate-then-measure. A tool drafts the candidate; the test set and the version comparison decide whether it earns the change. Neither half is optional — a draft you do not measure is a guess, and a measurement with no candidate to score is idle.

Grading by reliability-per-effort

The fourth step is grading: turning each run into a score against the criteria. The guidance for which grader to use is a single optimization — when deciding how to grade, “choose the fastest, most reliable, most scalable method.” [Official] Define success criteria and build evaluations · AnthropicT1-official original The score itself is “A score, generated by one of the grading methods discussed below” [Official] Building evals · Anthropic (2024)T1-official original — one part of an eval’s four-part anatomy (input prompt, output, golden answer, score), produced by comparing the output to the golden answer.

The methods rank by reliability-per-effort:

• Code-based grading comes first. It “is by far the best grading method if you can design an eval that allows for it” [Official] Building evals · Anthropic (2024)T1-official original — exact match, string-contains, or a regex over the output — because it is fast and highly reliable. If the criterion can be checked by code, check it by code; nothing else is cheaper or more dependable.
• Human grading comes next, for quality that code cannot capture. In the Console, this is concrete: quality grading lets you “Grade response quality on a 5-point scale to track improvements in response quality per prompt.” [Official] Using the Evaluation Tool · AnthropicT1-official original It is reliable but does not scale — a human caps the volume.
• LLM-based grading comes last, for judgement at scale. Its profile is “Fast and flexible, scalable and suitable for complex judgement. Test to ensure reliability first then scale.” [Official] Define success criteria and build evaluations · AnthropicT1-official original It can grade nuanced quality that code cannot express, across far more items than a human can — once you trust it.

That final clause — “Test to ensure reliability first then scale” — is the seam between this chapter and the next, and it is worth being precise about what it does and does not say. Here, the LLM judge is used: you pick it because the criterion needs judgement, you sanity-check that it agrees with you on a sample, and then you scale it across the set. That is the prompt-grading use of a judge. It is emphatically not a calibration project. Measuring the judge as an instrument — its agreement rate against human graders, its biases, the error bars on its scores — is a different and heavier discipline. This chapter only borrows the judge; ch23 calibrates it.

Quick reference

• The loop: define criteria → build test cases → iterate with tooling → grade outputs — then repeat. “This cycle is central to prompt engineering.” Define success criteria and build evaluations · AnthropicT1-official original
• Preconditions: criteria and tests are fixed before iterating — they are prerequisites, not by-products. Prompt engineering overview · AnthropicT1-official original
• Measurable criteria: “Use quantitative metrics or well-defined qualitative scales” Define success criteria and build evaluations · AnthropicT1-official original — multidimensional (accuracy, format, latency, cost).
• Eval-set tension: mirror the real distribution Define success criteria and build evaluations · AnthropicT1-official original and prioritize automatable volume — “More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.” Define success criteria and build evaluations · AnthropicT1-official original
• Iterate = generate-then-measure: the prompt improver drafts the candidate Console prompting tools · AnthropicT1-official original ; the Console eval tool versions and compares it. Using the Evaluation Tool · AnthropicT1-official original (Console UI is volatile — recheck after 2026-08-25.)
• Grading hierarchy: code-based first (the “best grading method if you can design an eval that allows for it” Building evals · Anthropic (2024)T1-official original ), then human (5-point scale Using the Evaluation Tool · AnthropicT1-official original ), then LLM-based (“Test to ensure reliability first then scale” Define success criteria and build evaluations · AnthropicT1-official original ).
• The ch23 seam: the LLM judge here is used, not calibrated — calibrating the judge as an instrument is the next chapter.
• Unit of analysis: a prompt. Switch to ch23 the moment the thing under test is an agent’s behavior over a task suite.

Practice