WhoWins: The First Line of a Prompt Wins Conflicts 72–95% of the Time

In real-world use, an agent receives commands from five places at once: the operator, the repository configuration, the user, tool outputs, and stored memory. These sources regularly contradict one another. Researchers led by Anish Laddha built WhoWins, a benchmark of 1,080 scenarios, and ran 14 open models through it. The result: across 1,887,356 scored responses, the outcome of a conflict depends heavily on where a line sits and noticeably less on who sent it.

Why Older Instruction-Hierarchy Benchmarks Hit a Wall

Existing instruction-hierarchy tests ask one question: does the higher-ranked source win? The WhoWins authors point to a methodological problem. A source determines three things at once: the text's position in the context, the message's role, and the label it arrives under. If the system prompt wins, it is unclear why. It could be because of its status, or simply because it comes first.

WhoWins therefore separates these factors. The benchmark has five task types, with up to five mutually exclusive instructions per scenario, and results are scored by code, without an LLM judge. Seven controlled experiments change only how the instructions are delivered. The authors tested 14 open-weight models, from 0.6B to 122B parameters, with thinking both on and off. They re-checked their main findings on held-out scenarios.

The

First Line Wins, but the System Role Does Not Guarantee Victory

Within a single message, the first of two conflicting lines wins in 72.42–94.66% of cases, depending on the model and mode. Across messages the effect is weaker: the earlier instruction holds in 22.95–62.65% of cases.

The most awkward finding for practitioners concerns the system level. A policy sent first in a system or developer message loses most conflicts to later lines in every model of the Qwen family. On gpt-oss the picture is reversed: there it wins most conflicts when the model takes action. In tasks involving tools and file writing, the same text placed first as a labeled user message wins more often than it does in a system message. This holds in every model and mode combination.

In other words, the "system" label does not translate into reliable priority. Much depends on how a particular model was trained to handle roles and how it behaves when it has to perform an action rather than just answer.

Conflicts Go Unnoticed

The authors also checked whether a model says anything aloud about the contradiction. By their keyword templates, no more than 2.92% of resolved responses to paired conflicts flag it or soften the wording. The agent picks one of the instructions and acts. The user and the operator never find out.

A separate part of the work deals with prediction. Can results from paired conflicts predict behavior in complex scenarios with several instructions? Predictions improve in every model and mode combination when the position of the line containing the task itself in the context is known.

What the Paper Contains

• The WhoWins benchmark: 1,080 scenarios scored by code.
• Five task types, with up to five mutually exclusive instructions per scenario.
• Seven controlled experiments that change only how instructions are delivered.
• 14 open-weight models from 0.6B to 122B, with thinking off and on.
• 1,887,356 scored responses, with findings checked on held-out scenarios.
• Single message: the first of two conflicting lines wins in 72.42–94.66% of cases.
• Different messages: the earlier instruction holds in 22.95–62.65% of cases.
• No more than 2.92% of resolved paired responses explicitly flag or soften the conflict.
• Paper length: 55 pages, 8 figures, 24 tables.

What to Change in Agent Systems Right Now

First: do not rely on the system label as protection for critical rules. If a safety policy must survive later user instructions or tool outputs, test it on the specific model, because Qwen and gpt-oss behave differently. Second: treat the order of lines in a prompt as part of the architecture, not the formatting. Third: for tasks involving tool calls and file writing, check priorities separately from chat scenarios, since the system policy wins less often there.

Finally, you need monitoring for silent conflicts. If an agent almost never signals a contradiction, you will have to find out about it from logs and tests. The code and data are available via pip install whowins, so you can run your own models through the same protocol.