Changing an agent’s interface can improve the actions available to it while making a trained model perform worse. A comparison that freezes the model measures the combination of the new interface and the model’s familiarity with it. It does not isolate interface quality.

An October 27 preprint by Jian Mu, Ruibo Shen and Yao Shu, affiliated with HKUST’s Guangzhou and Hong Kong campuses, studies this distinction. In a controlled MiniWoB++ experiment, reversing observation-field order preserves the optimal decisions but reduces a trained policy’s success from 85.0% to 19.2%. Other experiments examine adaptation after harness changes. Read the primary source.

What the evidence establishes

The authors distinguish the return the harness makes achievable from the gap caused by the policy’s decisions. This is a useful interpretation of their controlled setting, rather than a directly observable universal ceiling for arbitrary production software. Estimating an optimum with a stronger model also introduces a proxy whose limitations need to remain visible.

For evaluation teams, the practical implication is to keep two comparisons separate. One asks whether an existing agent survives a proposed interface change immediately. Another asks what can be achieved after comparable adaptation under each interface. A poor result in the first test need not settle the second question.

What changes in practice

That does not mean every regression should be excused as unfamiliarity. A revised harness may genuinely remove necessary information or make an action unsafe. Tests need to verify the environment contract, not merely train until the aggregate score recovers. Preserve action definitions, observation formatting and model revisions as separate artifacts.

The short-term contribution is a concrete warning about what frozen comparisons mix together. Longer-term, it could encourage evaluation designs that distinguish migration reliability from the potential of a new interface after training.

The evidence to watch

The next evidence is replication across broader tasks and realistic software changes, including revisions that make the best achievable behavior worse. This is a preprint, and the reported experiments have not been independently reproduced by this publication.

Source organization: HKUST (Guangzhou).

References
  1. Original research or engineering account

Questions, news tips or corrections? Contact [email protected].

Share this article

Post to XShare on LinkedInShare on WhatsAppEmail