What we found
The main things the results show, each on one page with the numbers behind it. They are rewritten from the results whenever a model is added.
- Finding 13.3×
Price doesn't buy quality
Kimi K3 costs $0.12 a task and got 69% of the work right. Claude Sonnet 5.5 costs 3.5¢ and got 70%. DeepSeek V4.1 Flash is within 5 points of the top for 0.43¢.
- Finding 28 of 12
It sounds right. That's the problem.
Beside the real documents public servants wrote and published, blind judges preferred the AI's version 8 times out of 12. Yet across the whole benchmark the best model got 30% of its work wrong. Good writing is no sign of a right answer.
- Finding 397% / 33%
"Best model" depends on the work
The best score on procurement evaluation is 97%. On ATIP redaction no model got more than 33% right.
- Finding 454% → 18%
Same model, different work, different answer
GPT-6 Luna got 54% of PubSec work right and 18% of Banking work. Muse Spark 1.3 scored 64% and 63%. A model's score on one kind of job tells you little about the next.
- Finding 54 of 17
We marked our own answer key
When three or more models gave the same answer and our key marked it wrong, we checked the key by hand. It was right 13 times out of 17. The other 4 times the models were. We fixed those and regraded everything.