A screenshot preserves layout. A sentence can preserve an instruction without the visual clutter around it. Neither is the universally best memory for a computer-using agent.
SwitchMem's October 13 study finds that preferred memory modality changes across interaction steps. Its router dynamically chooses textual or visual representations and improves on static memory strategies across the authors' three GUI benchmarks. Read the research paper.
Store alternatives, choose at use time
An agent locating a button may benefit from the previous screen. An agent recalling a completed form field may need a compact semantic record. Selecting one representation when memory is written commits every future query to that choice.
Keeping both formats introduces storage and selection costs. The relevant comparison includes those costs, rather than treating a router as free intelligence.
Test the system after interface redesigns, not only on repeated layouts. Visual memory can become misleading when controls move; text can become ambiguous when labels are duplicated.
The study supports adaptive representation within its tested workflows. Its wider implication is that memory quality is conditional on the next operation. The right question is not which format remembers more, but which format supplies the evidence this particular decision requires.
Questions, news tips or corrections? Contact [email protected].
