Rank model fallbacks by failure mode, not by score

Every model benchmark produces a table sorted by score, and almost every fallback chain is built by reading that table top to bottom. That is the wrong sort key for a chain whose job is to keep a production system correct when a provider misbehaves.

Two arms, close scores, different failures

On the 30 August 2026 long-form battery, two arms landed close: one at 16.3 of 40, the other at 19.3 of 40. The lower scorer’s failures were eight HTTP 429 rate limits. The higher scorer’s failures were six completions that came back as an empty string with a 200 status.

We placed the lower scorer higher in the chain. A 429 is loud, transient, and specific to that provider: the chain sees it, rotates, and the next arm answers. A silent empty is accepted as a success by any chain that does not check content, and it flows downstream into a parser, a template, or a published page.

The sort key

Rank arms on the pair (failure mode, score), with failure mode first:

1. Loud and transient (429, 503, timeout): costs one retry. Rank by score within this class.
2. Loud and persistent (400 on budget, 404 on a withdrawn model): costs the arm, but visibly. Fix or remove.
3. Silent (200 with empty or truncated content, a refusal dressed as an answer): costs correctness. An arm with any silent failures in three repeats goes below every arm with none, regardless of score.

Why this matters more than the number

A chain is only as safe as the worst thing it will accept. The score tells you how good the arm is when it works. The failure mode tells you what happens to your output when it does not, and that is the thing a production system is actually exposed to.

A shipped chain is a claim, not a state. Providers withdraw models without notice; on 29 August both arms of one of our live vision chains had been withdrawn, and the gate in front of them had been failing open. Poll the provider’s model list before trusting the chain, and re-run the battery whenever an arm changes under you.