We marked our own answer key
4 of 17
times the models were right and our answer key was wrong
A benchmark is only as good as its answer key, and ours had mistakes in it. After the 18 models ran the banking benchmark, we looked for answers where three or more of them agreed with each other and disagreed with the key, then worked each one out by hand from the task's own documents.
- 13
- 4
In 13 of the 17 we audited, the key was right: the models had all missed the same sentence. In 4, the models were right.
• A rule about small cheques was applied in one calculation and forgotten in the next, so four late cheques were missed.
• One made-up bank account had enough money in it when a not-enough-funds fee was charged.
• A deadline depended on a legal definition of "clear days" that we had not attached to the task.
• One policy's wording supported both the 2026 and the 2027 capital rules.
We corrected 13 answers across the 300 banking tasks and regraded every model from its saved work, without running anything again. No score moved by more than 4 points. Some models went down: they had matched the old, wrong key.
We run this check after every round now. A benchmark that never finds its own mistakes is not looking.