Free resources · For developers

Does Claude follow rules better when it can run a check?

A check script Claude can run got 59 of 60 replies to follow three rules. The same script, never mentioned, got 0 of 60.

Does Claude do better with a check it can run?

The docs say: "Give Claude something it can run that comes back pass or fail". We measured it with a rule a script can check: the format of a reply.

What was tested?

Three rules for each reply: first sentence 12 words or fewer, whole reply 80 words or fewer, and never write "I hope this helps". Claude Haiku answered the same 30 questions, twice each, so each setup has 60 replies. Its tool calls were logged. We graded replies with a second script that counts words differently from the check.

What happened?

Replies that followed all three rules, out of 60, by the second grader:

The check script in the folder, never mentioned: 0.
Rules told in the prompt only: 53.
Rules in the prompt, plus a check script Claude can run: 59.
The check script alone, with the rules not stated: 59.

In earlier tests with no tools, no rules scored 0 and rules in CLAUDE.md scored 43.

What did the check do?

With the check alone, Claude read the script in 60 of 60 replies and ran it in 60 of 60. The check said BROKEN at least once in all 60. With the rules also stated, the check said BROKEN in 47 of 60 and Claude ran it in 59 of 60. Claude never read or ran the script it was not told about.

What did it cost?

Rules in the prompt alone cost $0.0021 a reply. Rules plus the check cost $0.0031 and 3.8 turns. The check alone cost $0.0033 and 4.5 turns.

How far can this be trusted?

The check counts words, which is easy. Claude could read the script and see the rules.
One model, format rules only, and 30 questions we asked twice, which are not independent trials.
The earlier no-tools setups differ in more than the check.

What we would do

Write the check first, as a script that prints OK or says what is wrong.
Tell Claude to run it and rewrite until it prints OK.
Compare against the same questions with no check before you rely on it.

Check

Run your check on a reply you know breaks the rule. It must say BROKEN.
This result covers one model and format rules only.

Learn it step by step: Checking Claude's work