A benchmark name is not a complete experimental specification. Two evaluations can use the same tasks while giving models different tools, attempt budgets or feedback. Comparing the resulting scores as though only the model changed can lead to the wrong deployment decision.
The September 22 AISI and EvalEval account describes sharing evaluation results through Evaluation Cards and the Every Eval Ever reporting schema. The release accompanies research on inference compute and protocol, with configuration information for several benchmarks and models. It makes the conditions behind selected published results easier to inspect. Primary source.
Feedback changes the experiment
One distinction in the account is whether an attempt receives correctness feedback before another attempt. An oracle-assisted search measures something different from a model producing a single answer without that signal. More compute can improve observed task completion while changing the practical assumptions needed to obtain it.
The reporting layer cannot make unlike experiments identical. Its value is to expose those differences so a reader can decide which comparisons are meaningful. Missing configuration should remain missing rather than being replaced by assumed defaults.
Transcript information can also help explain a result, although public release needs to respect the content and permissions of the underlying tasks. A score alone rarely reveals whether failures came from reasoning, tools, time limits or the surrounding harness.
Better records support better decisions
Near term, evaluators can use structured records to preserve model versions, budgets and protocols alongside scores. Buyers can then ask whether the reported setting resembles the service they intend to run.
Over time, shared reporting could support more credible comparisons and meta-research if the important fields are consistently populated. The next evidence is wider adoption with complete records and reproducible reruns. The objective is not to eliminate disagreement about evaluations, but to make clear which experimental choices produced it.
Source organization: UK AI Security Institute. Source statements are attributed; operational implications and suggested evaluation criteria are editorial analysis.
References
Questions, news tips or corrections? Contact [email protected].
