One item, end to end
A single Business scenario from Round 1, shown exactly as an expert annotator receives it: the executive prompt, the graded checklist, six blind model responses, and the scores, rankings, and rationales of two independent experts. Responses appear blind, labeled A to F, until you reveal which model produced each one.
Executive scenario (prompt)
Graded checklist
- 1 States a clear recommendation (or a hybrid of two options) and explains why it rejects the other options with 1-2 reasons ● critical
- 2 Quantifies Italy's 5 year contribution to revenue and EBITDA (for the recommended option) and clearly states how it moves the company towards the board's target of 20% revenue growth and 7% EBITDA margin
- 3 Balances economic and social factors, specifically reputation and political risk from job reductions
- 4 Identifies the top 3-5 risks (union backlash, tech underperformance, political pressure, cost overruns, etc.) and gives a concrete mitigation action for each option)
- 5 Quantifies capex and EBITDA at the end of year 5 for chosen option (or hybrid option)
- 6 Estimates final capex for chosen option at the end of each 5 years and is broken down into major components (Italian store openings, Italian distribution network, German automation, etc. for whichever option is recommended)
- 7 Provides a comparison of labor costs impacts between entering Italy compared to a non-entry scenario, including change in total group labor cost
- 8 Compares revenue, margin, capex, labor costs, EBITDA of chosen option to non-entry alternative hypothetical
Model responses
Six models answered this prompt blind, labeled A to F. Read them, then flip the switch to reveal which lab wrote each.
How the two experts ranked it
Both experts scored every answer on the five-point rubric and produced a full preference order, blind and independently. On a hard item the two orders diverge — exactly what double-judging exists to surface. Experts stay pseudonymized (A04, A43).
See how items like this are built on the methodology page, or the field-wide scores on the leaderboard.