Does a check that runs on every reply beat a written rule?

A rule in CLAUDE.md held for 43 of 60 replies. With a Stop hook it held for 58 of 60, by a second grader.

Does a written rule hold?

A rule in CLAUDE.md is read by Claude, which then decides what to do with it. A hook runs every time, whatever Claude decided. We measured it. We asked Claude Haiku 30 software questions, twice each, under three reply rules. The rules: first sentence 12 words or fewer, whole reply 80 words or fewer, and never write "I hope this helps".

What happened?

Graded by a second checker that shares no code with the hook:

No rules: 0 of 60 replies followed all three.
Rules in CLAUDE.md: 43 of 60.
Rules plus a Stop hook that checks the reply: 58 of 60.
Rules plus a hook allowed to send the reply back again and again: 58 of 60.

The hook's own count said 60 of 60 for both. The two counted words differently, so we added a second grader.

What did the hook cost?

Average cost per reply was $0.0012 with rules alone and $0.0013 with no rules. With the hook it was $0.0015 (one retry) and $0.0014 (repeated retries). Average turns rose from 1 to 1.3 with one retry and to 1.18 with repeated retries. The hook sent 18 replies back in one setup and 11 in the other.

How far can this be trusted?

Each setup had 60 replies: we asked 30 questions twice. That is a small sample.
We tested only format rules, on one model. They are easy to count, and we did not measure whether the replies were good.
Claude Code ends a Stop hook that blocks eight times in a row. The script should let Claude stop if it already sent the reply back.

How do you add one?

Copy check.py from the kit into your project.
Add it as a Stop hook in .claude/settings.json, with the full path to the script.
Run the kit's test script on a reply you know breaks the rule. The checker must block it.
Ask 10 questions with and without the hook, and count the replies that break your rule.

We would put a rule that must never break in a hook, and a rule that needs judgment in CLAUDE.md.

Check

Write the rule as something a script can test. "Be concise" cannot be tested; "80 words or fewer" can.
This result covers one model, 30 software questions and format rules only.

Learn it step by step: What a hook is