Restricting retrieved documents to a historical date does not remove later facts stored in a model's weights. Yet finding those facts in a closed-book probe does not establish how they influenced an agent's investigation.
An October 13 study pairs historical company-comparison tasks with recall probes and identity interventions. It finds future-fact accessibility, but not the prespecified shift in evidence gathering toward the better-remembered company. Some observed patterns run in the opposite direction. Read the research paper.
Access, search and choice need separate measurements
The result resists an easy causal story. A model can know a future fact, allocate search differently and still change its final choice for another reason.
The controlled task set and unresolved mechanism limit generalisation. The study does not certify the absence of future leakage in other historical applications.
For point-in-time systems, audit external-source cutoffs, closed-book accessibility, search allocation and final outputs independently. Identity anonymisation can help test whether stored associations matter, but may also alter other context.
The insight is methodological: detecting contamination is necessary, while attributing its behavioural effect is another experiment. Treating the two as identical can produce confident explanations the observed agent behaviour does not support.
Questions, news tips or corrections? Contact [email protected].
