A research assistant can propose several plausible experiments without knowing which one deserves the next hour of compute. Judging the choice requires observing the alternatives that were rejected, not only the experiment the assistant chose to run.

University of Minnesota researcher Art Kanke introduces rTaste in an October 29 preprint. Six frontier models work from seven recent NLP papers, proposing minimum viable experiments, forecasting outcomes and ranking their proposals. A separate executor attempts all 415 experiments, and three model auditors assess them. The study finds useful proposals, but limited evidence that the agents’ rankings improve on choosing the first experiment they wrote. This is a bounded agent evaluation with model-based auditing, not a direct measure of scientific taste in general. Read the primary source account.

Running the rejected choices makes the comparison possible

If only a top-ranked proposal were executed, a successful result would say little about the ranking. Attempting every proposal creates outcomes for lower-ranked choices too. This allows the evaluation to ask whether the order supplied useful information beyond the initial generation process. It is a different question from whether an agent can implement an experiment whose goal has already been chosen.

The author reports 194 experiments judged sound, valuable and new, of which 73 are also judged beyond the seed paper and non-obvious. Those nested criteria should remain visible. Counting all 194 as strong new research directions would erase a material distinction in the study’s own interpretation. The auditors’ judgments also remain judgments by models under the stated protocol, rather than independently established scientific value.

Forecasting and prioritization need their own baselines

The forecasts beat a base-rate comparison, but the study does not show a measurable advantage over a fresh instance of the same model supplied with the paper and registrations. Teams cost more without a clear overall proposal-quality improvement. Neither finding establishes that collaboration or search can never help; they limit what this particular setup demonstrates about the additional work performed.

The immediate value is a released harness and dataset connecting registrations, forecasts and executed outcomes. A research group can examine prioritization separately from idea generation and implementation. That can improve budget decisions because a method producing many good candidates may still need a human or another process to choose among them.

The longer-term prospect is a research agent whose ranking reliably directs scarce resources toward informative tests. rTaste supplies a way to challenge that claim before a persuasive final report hides the rejected alternatives. The next useful evidence is reproducible ranking value against simple baselines, with stronger auditing and additional seed-paper domains.

Source organizations: University of Minnesota.

References
  1. Primary source account

Questions, news tips or corrections? Contact [email protected].

Share this article

Post to XShare on LinkedInShare on WhatsAppEmail