What's new
Every model we add, every change to how we test, and every correction, newest first. Also as a feed.
- · Correction
We audited our Banking answer key and fixed four errors
Where three or more models agreed on an answer our key marked wrong, we checked the key by hand. Of 17 such answers, the key was right 13 times and wrong 4 times. We fixed the four, regraded every model from its saved answers, and published the audit. Scores moved by at most 4 points.
- · New models
Eight more models on Banking
Gemma 4 26B and 31B, Mistral Small 4, Nemotron 3 Super, Ultra and 3.5 Lightning, Qwen3.6 35B and Qwen3.8 Flash. Eighteen models now have results on both benchmarks.
- · Method
Banking's standard run is now 60 tasks
Ten tasks from each of six kinds of banking work, chosen from the original 150. On the ten models that ran all 150, the shorter run changed no score by more than 3.3 points and costs 40% as much. Scores for single kinds of work now rest on 10 tasks each and have wide ranges.
- · New benchmark
GrindBench-Banking
A second benchmark: the everyday analyst work of Canadian banks and credit unions, from anti-money-laundering reports to capital ratios. Its ranking of models agrees closely with the public-sector one.
- · New models
Kimi K3.1 Pro on PubSec
Twenty-eight models now have public-sector results.