Two AI judges can disagree because they value style, detail or emphasis differently. Asking for a more elaborate rubric does not necessarily reveal those preferences.
PRISM's October 13 paper keeps evaluated content fixed while changing which attributes are emphasised. Human labels then train a correction from the judge's sensitivity profile. The authors report improved agreement with human rankings on summarisation and story evaluations. Read the research paper.
Calibration can use reactions instead of explanations
A judge may not accurately explain its implicit preferences. Controlled score changes provide another observable signal, without requiring its self-description to be trusted.
The method still depends on labelled human judgments and the domain in which the correction is learned. A story-evaluation calibration may not transfer to scientific evidence or software correctness.
Separate content-preserving probes from changes that actually alter the task. Otherwise a score movement could be appropriate rather than biased.
PRISM makes evaluator calibration more empirical. It asks how the judge responds to interventions, then checks whether those responses help predict disagreement. A judge's confidence in its rubric is less useful than evidence about how its scoring behaviour changes when the same work is presented differently.
Questions, news tips or corrections? Contact [email protected].
