A prompt comparison can appear to show a behavioral effect while changing more than one variable. If one wording is hostile and also asks for a shorter answer, a lower token bill does not establish that hostility caused the saving. The experimental control matters as much as the headline result.
Arize’s October 13 account tests prompt tone across six models and then revises the experiment to separate tone from brevity. The author reports that much of the apparent discount from rude prompts disappeared after removing length-related instructions. A further comparison holds the task sentence constant and changes only a short tone prefix. The findings are specific to the tested models and multiple-choice tasks. Read the primary source.
Visible output is not the whole bill
The source also records hidden reasoning tokens. A request for less visible commentary can shift computation into internal reasoning rather than simply removing it. Measuring output length alone would therefore misdescribe both cost and the mechanism behind a difference.
For teams running their own experiments, the practical change is to specify which variable is being tested and preserve the rest of the prompt. Accuracy, total billable tokens, latency and response behavior should be examined separately. Repeated runs and uncertainty are especially useful when differences are small or vary across models.
This study does not settle broader questions about how people should interact with AI. It provides a concrete example of a confound that can enter an evaluation unnoticed. The next evidence is replication on other tasks and production traffic with tone isolated from task instructions. Clear experimental design offers a more reliable optimization path than a provocative wording rule.
References
Questions, news tips or corrections? Contact [email protected].
