An automated research pipeline needs a way to reject ideas that merely restate prior work. Delegating that judgment to another language model creates a second research problem: whether the judge measures novelty consistently enough to guide the first system.

Ai2’s October 27 Old Ideas, Novel Problems release tests novelty judges with review-derived human ideas and a separate human-plus-generated setting. In one reported comparison, a prompt change moves GPT-5.4 accuracy from 93.5% to 40.9% on the same pairs. The generated-idea labels are acknowledged as weak assumptions rather than verified judgments for every item.

Read the original source and its reported conditions.

Keep the label construction visible

The study mines explicit reviewer statements about originality rather than treating acceptance alone as novelty. That creates a more relevant signal, but it still reflects a particular review process and research community. Results should not be promoted into a universal test of scientific importance.

The weakly labeled setting raises a second boundary. A disagreement with its label can reflect a judge failure, a weak label or both. Reporting setup sensitivity without relying on the labels provides complementary evidence: opposite verdicts for the same pair reveal instability even before correctness is settled.

More effort does not automatically fix the measure

The release reports that additional search and reasoning do not consistently improve the tested judgments. A larger evaluation budget therefore needs an explicit hypothesis about which error it should remove, followed by a controlled comparison.

The practical implication is to validate the novelty judge on the ideas it will actually receive and to test reasonable prompt variants. Automated triage may still be useful, but consequential rejection or ranking needs an inspectable evidence trail and a way to challenge the verdict.

References
  1. Ai2 source

Questions, news tips or corrections? Contact [email protected].

Share this article

Post to XShare on LinkedInShare on WhatsAppEmail