Free resources · For developers
Does a team of Claude reviewers find more bugs in code than one?
One auditor or three found 7 of 8 planted bugs in all 10 audits, and all 10 missed the same race condition.
Does splitting an audit across agents find more bugs?
Anthropic describes subagents as a way to "isolate context, run multiple analyses in parallel, and apply specialized instructions". We asked whether that structure improves a code audit.
What was tested?
We used a small Node app of 204 lines, with 8 planted bugs and 4 look-alike spots that are actually safe. The bugs include SQL injection, a delete route with no authentication, a fallback signing secret, and a stock update that is not atomic. We ran 5 audits with one general auditor, and 5 with three auditors focused on security, correctness and reliability. All used Claude Haiku. A fresh verifier then checked each finding.
What happened?
- All 10 audits found the same 7 bugs. None found the non-atomic stock update, a race that can oversell.
- No audit flagged a look-alike safe spot. The verifier kept all 7 real bugs each time.
- One auditor gave 6.8 other findings per audit. Three auditors gave 12.6.
- One audit cost $0.0367 with its verifiers. Three auditors cost $0.0596.
Every audit did report one real bug nobody planted: the order quantity is never validated, so a negative quantity raises stock.
What did the verifier do?
Across 97 other findings it confirmed 61 and refuted 36. On clear problems it agreed with itself: a login with no rate limit was confirmed 7 times and refuted never. On borderline issues it split. Tokens without an expiry claim were refuted 11 times and confirmed twice.
What does it mean?
More auditors gave more findings, not more of the planted bugs. The one bug that needed reasoning about two requests at once was missed by every setup. The verifier's verdict on a borderline issue changed between audits, so we treat it as a hint, and we think a person should decide.
How far can this be trusted?
- One model, one small app that fits in one context, and 5 audits per setup. Most bugs are textbook ones.
- "Subagents" here were separate headless sessions run by a script, not the Agent SDK's own feature.
- The other findings were not graded for truth. Matching findings to the key is by keyword. A review found wrong matches, and we corrected the key and regraded every audit.
What we would do
- Plant a few known bugs of different difficulty, including one that needs reasoning across requests, and see which audit finds them.
- Add focused auditors only if they find a planted bug the first one missed.
- We would keep a person in the loop, and read the verifier's reasons for refusals.
Check
- This result covers one model and one small application.
Learn it step by step: Orchestrator and workers
Sources
Sign up for membership to see the sources.