GrindBenchThe daily grind, benchmarked.
← All findingsFinding 2 of 5

It sounds right. That's the problem.

8 of 12
times blind judges preferred the AI's document to the one a person wrote

For 12 documents we had both an AI's version and the original a public servant wrote and published. Judges who didn't know which was which preferred the AI's version 8 times, the person's once, and called 3 a tie.

  • The AI's version8
  • The person's version1
  • A tie3
Which version blind judges preferred.

So the writing is good, from nearly every model. The answers are another matter. Across all the tasks, Claude Sonnet 5.5, the best model, got 70% right, and the model in the middle of the table, Nemotron 3 Ultra, got 53%.

That is why GrindBench doesn't mark on how a document reads. Every task has a checklist of what a careful manager would catch: the right figures, the right legal sections, nothing invented, nothing private left in. One deal-breaker fails the draft, however good the writing is.

Fluent writing used to be a sign that someone knew the subject. With AI it is free, so it tells you nothing. The only way to know a draft is right is to check it.

Try it: can you tell by reading? →

Next

"Best model" depends on the work

The best score on procurement evaluation is 97%. On ATIP redaction no model got more than 33% right.