Same model, different work, different answer
54% → 18%
GPT-6 Luna on PubSec work, then on Banking work
18 models have run both benchmarks. Most land in a similar place on each. GPT-6 Luna does not: 54% of its PubSec work was right, and 18% of its Banking work.
- 54%
- 18%
- 64%
- 63%
Banking tasks lean on long, exact sums: totals over dozens of rows after the exceptions are taken out, where being close is still wrong. Muse Spark 1.3 held steady (64% and 63%).
Writing more doesn't fix it. Nemotron 3.5 Lightning wrote about 38,000 tokens of working for each Banking task, most of it reasoning nobody reads, and got 2% of them right.
A benchmark score describes the work that was tested. It is a poor guide to work that was not.