Free resources · For businesses

Can a second chat catch wrong facts in a client summary if it has the contract?

A reviewer with the contract caught 30 of 30 planted changes on one model and 29 of 30 on the other. A reviewer without it caught 6 and 0.

Can a second chat catch wrong facts in a summary?

A summary sent to a client can hold a wrong number. Claude can "verify each claim by finding a supporting quote after it generates a response". We asked: does a second chat catch wrong statements in a client summary, and what does it need to do that?

What was tested?

A made-up services agreement of 4,867 words, with amendments that change earlier numbers. Claude Haiku 5.5 and Claude Haiku 4.5 each wrote 5 one-page summaries for a finance director. We then changed three facts to wrong values in each of 10 summaries, such as 45 days to 30 days, or Oregon to Washington. That makes 30 planted changes. A fresh reviewer, run on each model in turn, listed every statement it judged wrong. One kind of reviewer also had the contract, and the other had only the summary.

What happened?

Planted changes caught, out of 30:

Reviewer with the contract: 30 on Haiku 5.5, and 29 on Haiku 4.5.
Reviewer with only the summary: 6 on Haiku 5.5, and 0 on Haiku 4.5.

Haiku 4.5 said it could not check without the contract, in all 20 of its reviews. Haiku 5.5 caught its 6 by spotting statements that contradicted each other.

What did it cost?

A review with the contract cost $0.0057 on Haiku 5.5 and $0.0594 on Haiku 4.5. Without it, $0.0026 and $0.0167.

What did the reviewer find on its own?

One summary, by Haiku 4.5, said the data breach liability was "capped at 2× monthly fees". The contract says two times "the amount in Section 9.1", which most plausibly means the general cap on the fees of the prior 12 months. The reviewer with the contract flagged it. The reviewers also listed many smaller points on unaltered summaries, some real and some nitpicks.

How far can this be trusted?

One made-up contract, 10 summaries and two small models. The contract has conflicts created by its own generator.
Planted changes were simple swaps of a value. We did not test missing or vague statements.

What we would do

A model scores its own output higher than others do.

Give the reviewer the source document, not only the summary.
Use a new chat, and paste this prompt:

Check the summary against the contract. List every statement that is wrong, each on its own line as WRONG: the incorrect text -> what is right. If every statement is correct, write only Verdict: OK.

Check each listed point yourself before you send anything.

Check

Planted errors: change three facts in one of your own summaries and see whether the reviewer finds them.
This result covers one contract and two small models.

Learn it step by step: A separate reviewer