It sounds right. That's the problem.
8 of 12
times blind judges preferred the AI's document to the one a person wrote
For 12 documents we had both an AI's version and the original a public servant wrote and published. Judges who didn't know which was which preferred the AI's version 8 times, the person's once, and called 3 a tie.
- 8
- 1
- 3
So the writing is good, from nearly every model. The answers are another matter. Across all the tasks, Claude Sonnet 5.5, the best model, got 70% right, and the model in the middle of the table, Nemotron 3 Ultra, got 53%.
That is why GrindBench doesn't mark on how a document reads. Every task has a checklist of what a careful manager would catch: the right figures, the right legal sections, nothing invented, nothing private left in. One deal-breaker fails the draft, however good the writing is.
Fluent writing used to be a sign that someone knew the subject. With AI it is free, so it tells you nothing. The only way to know a draft is right is to check it.