A robot's memory should help it decide what to look at next

A camera-equipped agent faces an unusual information problem: it can sometimes change the evidence available to answer a question. A language model reading a document cannot walk around the document to reveal its hidden side. An embodied system can turn, move and inspect, provided it recognises that its present observations are inadequate.

This makes memory and action tightly connected. A useful memory can prevent unnecessary movement, but it can also expose an unanswered question that only movement will resolve. MemAct studies that joint decision, bringing retrieval, observation and stopping into the same loop instead of evaluating understanding as a passive stream of incoming frames.

A training set with different evidence needs

MemAct, posted October 13, combines structured memory, retrieval of past frames and active movement within one vision-language reasoning loop. Its training and benchmark suite includes questions requiring current observations, historical evidence or exploration, plus a separate timing set. Read the research paper.

The authors describe approximately one million training examples spanning spatial reasoning, memory writing, retrieval and exploration. Their main benchmark contains 20,000 question-answer pairs, with another 2,000 timing questions evaluated separately. The categories distinguish questions answerable from the current observation, questions requiring past evidence and questions requiring exploration.

That separation is valuable because identical answer accuracy can conceal very different behaviours. An agent may answer correctly because it recalled an earlier view, because it explored effectively or because the question allowed a plausible guess. A benchmark that distinguishes these cases provides a better basis for diagnosing what has improved.

Timing questions probe a different memory skill

MemAct’s separate timing set acknowledges that remembering what appeared and remembering when it appeared are not identical. A stored frame can accurately show a desk while giving the wrong answer about whether the scene changed before or after another event. Streaming evidence includes order as well as content.

That distinction matters when an agent chooses whether to retrieve or explore. If the question concerns the current state, a historically correct observation may still be insufficient. If it concerns a past state, moving the camera now cannot recreate the missing history. The action loop must therefore interpret the time reference of the question before deciding what evidence to seek. The paper’s suite makes temporal understanding visible as a separate category; it should not be collapsed into a generic spatial-answer score. An agent can become better at recognising rooms without becoming equally better at locating events within their history.

Movement has to earn its cost

MemAct combines supervised fine-tuning with counterfactual action optimisation, incorporating cumulative evidence gain, action cost and final answer quality. The design recognises that more observations are not automatically better. A long detour can reveal information while still being a poor decision for the task.

Its reported comparisons support the full approach on the authors’ suite. They do not establish that an evidence-gain objective automatically handles stairs, fragile objects, moving people or every other constraint that physical exploration introduces. The information objective and the execution envelope remain distinct engineering responsibilities.

The valuable answer may be a request for a new view

For an assistant inspecting a room, uncertainty can be actionable. It might know that a cupboard was visible earlier, but not whether its contents have changed. The appropriate next step could be to revisit it, rather than to produce a more elaborate answer from the old frame. Memory is helping precisely because it identifies what is missing.

This changes what a successful spatial model looks like. It should be assessed on the quality of its evidence-gathering decisions as well as the fluency of its final response. The larger opportunity is a system that spends physical effort deliberately, rather than treating every unseen fact as a failure of recall.

Questions, news tips or corrections? Contact [email protected].