A game agent completes a harder task after practice. Did it acquire a skill, or simply keep tools, materials and a favourable world state from the earlier run?
MC-EvoBench, posted October 9, resets target worlds and allows only a declared agent artifact to cross from source to target. Related, unrelated and shuffled experience conditions receive matched budgets. Its reported transfer findings are limited and unstable across some probes. Read the research paper.
The boundary must exclude accidental help
World resets separate durable capability from inherited resources. Matched exposure helps distinguish useful experience from extra time spent interacting.
Stage-level audits reveal another trap: an early bottleneck can prevent later abilities from being observed. Failure to craft a basic item says little about whether the agent could solve a later planning problem if it reached it.
The benchmark concerns a controlled Minecraft setting, not physical robot transfer directly. Its evaluation design is nevertheless relevant to embodied self-improvement claims.
Specify what persists, reset everything else and measure stages separately. A memory file is evidence that something was saved. Transfer to a clean task is the additional evidence needed to call that saved artifact a reusable capability.
References
Questions, news tips or corrections? Contact [email protected].
