Flattening a grid into tokens scatters some neighbouring cells far apart in the sequence. A model learning a local update rule must recover those relationships before it can learn the rule itself.
An October 6 paper proposes spatial induction heads: one layer gathers the relevant neighbourhood, and another matches that configuration to earlier occurrences. Experiments on cellular automata show generalisation to unseen rules, alongside attention patterns consistent with the proposed mechanism. Read the research paper.
Learning the rule starts with routing the evidence
The result separates two abilities often mixed together in discussions of in-context learning. A model must identify which observations belong together, then infer how they predict the next state. Failure at the first stage can look like failure to learn from examples.
Cellular automata offer unusually clean local structure. Natural scenes contain occlusion, ambiguous objects and changing viewpoints, so the demonstrated circuit is not an explanation of all visual reasoning.
It nevertheless suggests an informative diagnostic. Change the serialisation while preserving the underlying spatial problem. If performance drops, the bottleneck may be evidence routing rather than lack of task knowledge.
For multimodal models, presentation is part of learning. A few-shot example is only helpful when the architecture can bring its relevant pieces together at the point where a prediction is made.
References
Questions, news tips or corrections? Contact [email protected].
