Systems audit
Model rankings do not transfer between prompts
Published 28 Oct 2026 · measured 2026-08-29 · Darko AI engineering
Run-to-run spread on identical input was about six points; one arm scored 13 and 29 on the same prompt. One sample is not evidence.
When we benchmark a chain for a client we are often asked why we do not start from a published leaderboard. The honest answer is that we tried, and the numbers did not survive contact with a real prompt.
What the same model did on two prompts
In the August 2026 batteries one arm scored 35 of 36 on one pipeline’s prompt and 14 of 42 on another pipeline’s prompt, run on the same day with the same settings. Both prompts were production prompts: real system instructions, real input shapes, real output schemas. The model had not changed. The task had.
Leaderboards aggregate across tasks that are not yours. A model that writes excellent short-form copy may truncate a long structured answer, or follow a schema loosely, or refuse a category of input your pipeline depends on. None of that shows in an aggregate score.
What one sample is worth
On identical input, the run-to-run spread for a single arm was about six points on a 36-point scale, and one arm scored 13 and 29 on the same prompt in two runs. A single sample tells you where the arm landed once, not where it lives.
So the minimum is three repeats per arm per prompt, and the figure we rank on is the mean with the spread shown next to it. An arm with a mean of 28 and a spread of 3 is a different proposition from an arm with a mean of 28 and a spread of 16, and the second one is the one that will embarrass you on a Tuesday.
The procedure we run
- Take the pipeline’s actual prompt, with a real input. Not a simplified version.
- Run every candidate arm three or more times at a completion budget of at least 32,768 tokens, so budget failures do not masquerade as quality failures.
- Score with the same rubric the pipeline’s own quality gate uses, so the benchmark measures what the gate will enforce.
- Record failure modes separately from scores, then rank by failure mode first.
- Keep the table with the prompt and the date. When a provider changes a model under you, re-run it; the old table is history, not state.
A ranking is a measurement of one prompt on one day. Treat it as anything more and the chain you ship is built on someone else’s task.