An assistant answering “I have forgotten that” has made a claim, not demonstrated deletion. The information may remain extractable, or the system may have removed unrelated memories as well.
Bemula's October 13 paper pairs dialogues with and without forget requests, then tests removal, preservation, relearning and resistance to extraction. The evaluated memory systems perform poorly under its demanding end-to-end criterion. Read the research paper.
Forgetting needs a control condition
A system should not receive credit for failing to recall something it never stored. The paired version establishes that the target was available before testing whether the request removed it.
Selective removal is also different from resetting the whole memory bank. Users asking to delete one outdated fact still expect other useful context to survive.
The benchmark is small and controlled; its results are not a comprehensive audit of every product implementation. They nevertheless identify a capability ordinary recall leaderboards omit.
For agent builders, test indirect questions and derived summaries after deletion, then ask unrelated personal questions. A correct forgetting mechanism must change the requested information's availability while preserving the rest of the relationship. A conversational acknowledgement is the beginning of that process, not its evidence.
References
Questions, news tips or corrections? Contact [email protected].
