A search result list can look diverse while returning several copies of the same passage. When those copies occupy limited ranking positions, a retrieval metric measures both the model’s choices and an artifact of the corpus. Correcting the data changes the meaning of a score, even if the retrieval code stays fixed.
The University of Innsbruck team’s current RETECO training and development card documents a v1.1 deduplication of the temporal-retrieval corpus. It provides an old-to-retained document-ID map and keeps the earlier revision available. The card explicitly distinguishes these public development materials from the separate hidden SemEval test set. The revision is reviewed here as available on October 27; no new release date is inferred. Read the original source.
Make corpus versions visible
The correction makes versioning part of evaluation. A ranking produced against the earlier corpus may contain several identifiers for equivalent evidence. Comparing it directly with a ranking over deduplicated text can attribute a change in the benchmark to a change in the model. The migration map provides a way to inspect that distinction rather than conceal it.
Temporal retrieval adds another constraint: two passages that look similar can refer to different versions or periods. Deduplication must therefore preserve distinctions that matter to the question. An evaluator should record the exact transformation rule and verify that a relevant date or number has not been erased by an overly broad similarity threshold.
A development set is still development
The immediate task for researchers is to pin the corpus, judgments and scoring code together. The split name alone is insufficient if its underlying documents changed. A result table should make clear whether earlier runs were remapped, rerun or left on their original revision, and a claimed improvement should use comparable conditions.
The public development split also remains a development resource. Repeatedly selecting methods against its judgments can teach the research process to that split. A private final evaluation addresses a different question from whether a pipeline can reproduce a published baseline. Neither should be presented as an operational guarantee for a customer’s document collection.
Separate corrections from progress
A cleaner corpus can make differences between retrieval methods easier to interpret. In the longer term, explicit migration records could make benchmark maintenance less disruptive by separating data corrections from modeling progress. That depends on preserving access to older revisions and explaining which scores remain comparable.
What to watch is a versioned evaluation record that shows the same systems before and after the correction. Such a record would establish how much of the apparent change belongs to the corpus and leave the remaining model improvement open to proper measurement.
Questions, news tips or corrections? Contact [email protected].

