A tool argument may be correct because it follows from a default or several earlier facts. A judge receiving only the final request may reject it because the derivation is absent.
An October 13 study calls this the derivation gap. Controlled source relocation shows judges favouring visible references, including incorrect ones. Its DeRef method constructs a reference from the full history without seeing the candidate argument. Read the research paper.
Verification needs independent access to evidence
The candidate-blind step matters: a reference writer shown the proposed answer could rationalise it rather than derive an independent expected value.
The reference itself can still be wrong. Improving agreement with a derived reference is not the same as making every tool call correct, and full-history processing has a cost.
Test defaults, multi-step derivations and deliberately misleading visible values separately. A judge calibrated only on explicit answers may perform poorly when the application depends on inference.
The paper exposes a limitation in outsourced verification. The second model is not automatically an independent check if it lacks the evidence needed to reproduce the decision. A useful judge needs more than authority to approve; it needs the informational basis to disagree correctly.
References
Questions, news tips or corrections? Contact [email protected].
