Benchmark examples
Real tasks from the benchmark, with everything the models were given. Open one to read what each model wrote and see exactly what the grader checked, what it passed, and what it caught.
These come from the practice set we use to build and test the grader. The tasks that decide the rankings stay private, so models can't be trained on them.
- Performance analysis · Alberta · English
Annual report: transportation results
Write the results discussion for six performance measures in Alberta Transportation and Economic Corridors' 2024-25 annual report, comparing each result with its target and with earlier years.
✓✓✓✓4 of 4 acceptable - Performance analysis · Alberta · English · has a gap to flag
Annual report: results with conflicting figures
The same annual report discussion, but the documents disagree on the 2022-23 collision rate (46.5 in the business plan, 46.6 in the results data); a good answer flags the conflict and says which figure it uses.
✗✓✓✗2 of 4 acceptable - ATIP redaction · Canada · English
Access to information: redact an email
Prepare a short internal email for release under the Access to Information Act: black out the personal and confidential details, cite the exemption for each, and leave everything else visible.
✗✓✓✓3 of 4 acceptable - ATIP redaction · Canada · Français
Access to information: redact an email (French)
The same access to information redaction, done on the French version of the email, with the answer in French.
✗✓✓✓3 of 4 acceptable - Briefing & QP notes · Canada · English
Question Period note: farm income
Update a Question Period note for the Minister of Agriculture on farm income in 2024 and 2025, from Statistics Canada's latest releases and the previous note on the issue.
✓✓✓✓4 of 4 acceptable - Briefing & QP notes · Canada · Français
Question Period note: farm income (French)
The same farm income Question Period note, written in French from the French versions of the documents.
✓✓✓✓4 of 4 acceptable - Briefing & QP notes · Canada · English
Question Period note: job vacancies
Update a Question Period note for the Minister on Statistics Canada's job vacancy figures for the second quarter of 2025, saying which figures are seasonally adjusted.
✓✓✓✓4 of 4 acceptable - Briefing & QP notes · Canada · Français
Question Period note: job vacancies (French)
The same job vacancy Question Period note, written in French from the French versions of the documents.
✓✓✓✓4 of 4 acceptable - Performance analysis · Edmonton · English
311 requests: 2025 compared with 2024
Write a short briefing memo for Edmonton's Executive Committee comparing 311 requests in 2025 with 2024, from the City's open data.
✓✓✓✓4 of 4 acceptable - Performance analysis · Edmonton · English · has a gap to flag
311 requests by ward
Write a short memo comparing 311 requests across Edmonton's wards; the catch is that about 60% of 2025 requests have no ward recorded, which a good answer flags before ranking the wards.
✗✗✗✗0 of 4 acceptable