Putting many items in one prompt can reduce the cost of automated grading. It also places each item beside other texts and earlier model verdicts, creating context that a single-item evaluation never sees.
Peng Xie and Amr Alanwar of the Technical University of Munich test this assumption in an October 29 preprint. Across harm assessment, short-answer grading and review ratings, they regroup the same 2,000 items into batches of 25. Five open-weight 4–9B models change 11–22 percent of harm and grading decisions between random groupings; four larger 14–32B models change 6–15 percent. Aggregate discrimination can remain close to single-item grading even while individual decisions move. Read the primary source account.
Similar totals can hide different treatment of an item
An accuracy or AUROC summary can remain relatively stable when some items move up and others move down. That does not make the item-level changes irrelevant to a person receiving a grade or a moderation decision. The study therefore measures decision flips alongside aggregate quality, keeping two questions separate: how well the method ranks a population, and how consistently it treats a particular example.
The analysis distinguishes the item’s position, the order of its neighbors and which neighbors share the prompt. This helps identify why simply voting over rearrangements of the same batch cannot remove every effect. The authors report that nine-permutation majority votes still leave 52–60 percent of the flips in their experiments. Those percentages describe the residual instability under the tested procedure, not the fraction of all decisions that are wrong.
A serving optimization needs a judgment-level check
The easier star-rating task shows fewer changes than harm or short-answer grading. That difference limits broad claims about batching: task ambiguity and decision thresholds matter. It would be equally misleading to say batching is always safe because one aggregate score barely changes, or always unusable because a harder task exposes sensitivity.
The immediate implication is to compare the receiving workload with single-item grading before accepting a cost saving. A useful test retains item identifiers and measures disagreement after regrouping, rather than reporting only mean accuracy. Cached common instructions may recover some cost without putting unrelated items in the same judgment context; the paper proposes this direction, rather than proving a universal production configuration.
Batched inference can remain valuable where its behavior matches the task’s tolerance for instability. The broader contribution of this work is making an otherwise hidden serving assumption measurable. The next evidence is a task-specific agreement record that survives changed groupings and preserves meaningful individual decisions, with aggregate metrics reported alongside it.
Source organizations: Technical University of Munich.
Original research evidence

References
Questions, news tips or corrections? Contact [email protected].


