How do you test a Claude workflow, not just one answer?

Twenty to fifty real failures, graded the same way every time. Why one passing run proves little.

The idea

You changed a prompt, a skill or a CLAUDE.md, and you want to know if it helped. One good answer cannot tell you. Test the workflow, not one answer. A small set of tasks, graded the same way every time, shows whether a change helped.

How to start

Take 20 to 50 simple tasks from real failures.
Grade with code wherever you can, because it is the fastest and most reliable way.
Write each task so two experts would reach the same verdict. If a person could not pass it, rewrite it.
Read the transcripts to check the grading is fair.
Rerun the set after every change, so the check runs each time the work changes.

Why one run is not enough

If a workflow passes 75% of single runs, three runs in a row all pass only about 42% of the time. Run each task more than once. Measure how often all runs pass, not just how often one does.

Three kinds of grader

Code: fast, cheap and objective, but brittle when an answer is valid in an unexpected form.
A model: flexible and good for open-ended answers, but it needs calibration against a person. A model judge has known biases, such as favouring its own output. Use a different model to judge than the one that wrote the answer.
A person: the gold standard for complex tasks, but slow and costly. Use it to calibrate the others.

Two kinds of test set

A capability set asks what the workflow can do well and starts with a low pass rate. A regression set asks whether it still does everything it used to, and should sit near a 100% pass rate. When a capability task passes reliably, move it to the regression set.

Watch for

Grading that rejects a valid answer, such as “96.12” against “96.124991”.
Tasks so vague that a model can fail through no fault of its own.
Leftover files or cached data that make results look better than they are.
An agent that takes a route the test did not expect. Read the transcript to see whether it is a real failure or a better answer.

The same rule for skills

Build the test before the skill. Run Claude on real tasks without the skill, write three scenarios from the gaps, and measure the baseline. Then write only enough instructions to close them.

Check

Run the set before and after one change. Trust the change only if the grade moves the way you expected.
Run each task more than once. The share where all runs pass should be lower than the share where one run passes.
Read the transcript of each failed task. Confirm the model made the mistake, not the grader.

Learn it step by step: Write the evals first