A classifier scoring roughly half the answers correctly on a balanced binary task may appear weak. It may be doing something worse: giving the same label regardless of the question.
An October 12 study documents label collapse across tested autoregressive and diffusion language models at the 8–9B scale. It finds that demonstration arrangements and label words can change the behaviour, even when class counts are balanced. Read the research paper.
Inspect dependence on the input
Average accuracy cannot distinguish a query-sensitive but confused classifier from a constant response on a balanced test. Those systems have very different prospects for improvement. More examples might help the first and leave the second unchanged.
The authors propose examining separation between label probabilities alongside accuracy, with care around tokenisation and output formatting. In practice, also pair contrasting queries and ask whether predictions move when the relevant evidence changes.
The result is specific to tested models and arrangements. It does not establish that all in-context learning collapses. It does establish a diagnostic gap: balanced evaluation can make a pathological shortcut look like ordinary uncertainty.
Before tuning a few-shot prompt, verify that the model is responding to the input at all. Otherwise an apparently modest accuracy problem is being mistaken for a functioning decision process.
Questions, news tips or corrections? Contact [email protected].
