Lesson 3 of 6
Use a council of models
When I have a decision that matters, I give several models the same question and brief and ask them to answer separately. I compare their reasons, especially where they disagree, before deciding what to do.
Chris Blattman describes how he uses separate agents to develop a plan and how he brings in fresh reviewers after giving his own judgement. Blattman’s council guide
John H Boyer’s instruction files for his Greek readers require work by a Claude agent to be reviewed by GPT or Grok, and work by Codex to be checked by Claude before it is committed. They record that Grok’s content checks twice found classes of defect that the other models had missed. This comes from his project’s instruction files.
Two papers describe limits. A primer on agentic AI systems for scientific discovery in Nature Reviews Methods Primers (Feng and colleagues, published 22 October 2026) lists multi-agent coordination among its failure modes: adding agents can make results worse. Feng and colleagues’ primer Lu and colleagues’ paper in Nature on automating AI research (March 2026) uses an automated reviewer built from five reviews by the same model plus a meta-review. Because one model wrote all five, they share its biases, and the paper does not test whether the reviews are independent.
Earlier in 2026, Matthew Schwartz supervised Claude Code through a quantum field theory calculation and published his account on 23 March 2026 as a guest post on Anthropic’s website. The project plan was built by merging outlines from three models. He reports that GPT and Claude caught each other’s errors, and that GPT solved the hardest integral, which Claude then incorporated. This is his own report of one project, hosted by the vendor.
Tiago Ventura’s slides for a faculty retreat in 2026 report that cross-checking with a second model improved a long instruction file that Claude Code had mostly written. He also reports running a tournament across models among the top candidate papers for a replication, on the grounds that different models have different evaluation biases. The slides are terse bullets, so there is little detail.
Try a council
Pick a decision about a research design or an interpretation. In fresh sessions, give each model the same question, evidence and criteria. Ask each to recommend an approach and explain what would change its recommendation. Keep the first answers separate. When you compare them, look for the assumptions or evidence that account for any disagreement. If you have access to one model, Blattman’s starter kit shows how to give separate runs different reviewing roles. Blattman’s starter kit
My agents do this through my model council skill. It asks for my approval first, gets separate answers from other models, and keeps the disagreements and the evidence behind any recommendation.
Models can make the same mistake. Check any claim that affects your decision against the source or data, even if the models agree. For a result you can test, start with a known result and keep an independent check.