Free resources · For developers

How many test cases do you need to tell two setups apart?

A simulation on 240 real graded replies shows how many cases it took to tell two setups apart. 20 to 50 tasks is the usual advice.

How many cases do you need to start?

The usual start is "20-50 simple tasks drawn from real failures". In early agent development, a "large effect size means small sample sizes suffice". More mature agents "may need larger, more difficult evals to detect smaller effects".

What was simulated?

We used 240 real graded replies from Claude Haiku. Each had either followed three reply rules or not. We took two pairs of setups and drew their true pass rates from what we observed. Then we drew 5, 10, 20, 30 or 50 cases and counted how often the better setup scored strictly higher. A big gap: rules in CLAUDE.md, 43 of 60 followed, against rules plus a Stop hook, 58 of 60. A small gap: a 201-line CLAUDE.md, 40 of 60, against a 5-line one, 51 of 60.

What happened?

The better setup scored strictly higher in this share of trials:

Big gap: 73 percent with 5 cases, 87 with 10, 95 with 20, 98 with 30, and 99 with 50.
Small gap: 61 percent with 5 cases, 74 with 10, 83 with 20, 88 with 30, and 93 with 50.

Ties are not wins, and not wrong answers either. With 5 cases the big pair tied 22 percent of the time and the small pair 24 percent.

What does it mean?

At a bar of 90 percent, the big gap needed 12 cases and the small gap needed 37. The bar is a judgment, not a rule. Anthropic's 20 to 50 is enough for both, though the small gap needs the upper half of it. The small gap needed more cases than the big gap.

How far can this be trusted?

Two comparisons, one model, one kind of rule. Every case came from the same pool, so none was harder than another.
The 90 percent bar and the starting assumptions about the pass rates are choices, not measurements.
Every case was drawn from one pool, so the test does not copy how real failures differ.

What we would do

Start with 20 to 50 cases from your own real failures.
Run the calculator in the kit with your two scores. It prints the smallest number of cases that clears the bar.
If you cannot afford that many, make the two setups differ more, or accept that a small score difference is not reliable.

Check

Put your own pass counts into the calculator, and keep the setups you compare as close to identical as you can.
This result covers one model and one kind of rule.

Learn it step by step: Write the evals first