A user can change a requirement after an agent has started working. They may correct a mistaken assumption, supply missing information or withhold authorization for the next action. A benchmark that gives every instruction upfront cannot fully test that relationship. The final file may look correct while the process ignored an intervention that would matter in actual use. Collaboration needs evidence beyond the state of the finished artifact.
RealCLI, an October 15 preprint from researchers including IQuest Research, Beihang University and Renmin University, pairs task execution with an interactive setting. One mode supplies the specification upfront; the other uses a scripted user to introduce information and decisions during execution. The framework covers five collaboration dimensions, with twenty-two sub-dimensions and fifty-one scenario templates. The authors report a best joint collaboration score of 37.6 percent under their evaluation. Read the primary source account.
Paired tasks make the timing visible
Giving the same broad task in two modes helps expose the cost of interacting as requirements evolve. The comparison is not simply between an easier prompt and a harder prompt: it asks whether an agent can incorporate new information while preserving progress. For a developer, that can identify weaknesses that remain hidden when all constraints are present at the beginning of a run.
The scripted user makes the experiment repeatable, while setting a boundary on what it represents. Real people vary in how clearly they express corrections, when they interrupt and which omissions they notice. A benchmark can isolate selected situations without reproducing that full distribution. Its results should therefore describe performance under the supplied interaction protocol, rather than a universal measure of how pleasant or trustworthy an agent is to work with.
Finishing the work and respecting the interaction can diverge
An execution score asks whether the requested end state was achieved. A collaboration measure can also ask whether the agent stayed understandable, accepted steering, calibrated its claims and handled authorization appropriately. Those objectives overlap, but neither implies the other. A system may achieve an end state through actions the user did not permit, or communicate well while failing to complete the work.
Keeping those outcomes separate is useful for diagnosis. A single aggregate can obscure a model that succeeds on routine execution but consistently mishandles corrections. Conversely, a cautious system might score poorly on completion while demonstrating stronger boundary management. The practical goal is to understand the trade-off and improve both behaviors, rather than use one favorable number to dismiss the weakness revealed by the other.
Interaction quality depends on the surrounding system
The model is only one part of a command-line agent. Tools, retained context, interruption handling and the way user messages enter the workflow can affect the observed result. Reproducing RealCLI therefore requires the harness and configuration as well as the checkpoint. A comparison that silently changes those components risks attributing an interface improvement or regression entirely to the model.
The benchmark’s scenarios can also help teams build application-specific evaluations. The important step is to preserve the actual decision the user needs to control. A correction about a filename and an authorization decision about an external action carry different implications. Testing both under an undifferentiated instruction-following label can conceal whether the system understands what changed and which work remains authorized.
A useful failure trace would show the user’s intervention, the state of the task when it arrived and the agent’s subsequent actions. That sequence distinguishes a failure to notice new information from a failure to act on information that was understood. It also reveals whether an agent unnecessarily abandons work that remains valid after a correction. Those are concrete engineering questions that a final pass-or-fail artifact cannot answer alone, and they explain why retaining interaction records is central to this kind of evaluation.
The near-term contribution is a reproducible way to ask a broader question of CLI agents: can they complete realistic work while remaining responsive to the person directing it? The longer-term value depends on whether benchmark improvements transfer to less scripted interactions. Independent replications and carefully bounded real-user studies would strengthen that connection. Until then, the reported scores establish a gap under the authors’ paired protocol and provide a concrete target for improvement, without settling every aspect of collaborative agent quality.
Source organizations: IQuest Research.
References
Questions, news tips or corrections? Contact [email protected].


