How EuroExec is built
Every item moves through the same pipeline, from an expert's first-hand scenario to a gold record that two reviewers have judged and signed off. Authoring, automated screening, and human grading are deliberately kept apart, so quality comes from the process itself rather than a check at the end.
The pipeline, step by step
Expert authoring
46 vetted domain experts (CV screen plus a one-on-one interview) each write a hard, self-contained executive scenario and a 5–10 point grading checklist, drawn from problems they have worked first-hand. Copy-paste is disabled in the tool, so every prompt is original and no pasted AI text slips in.
Automated QA
An LLM screens each item on six criteria. Blocking: hint leakage, checklist relevance, prohibited patterns. Warning: professional language, a clear role, concrete specificity. Anything that fails goes back to the author with structured notes.
Difficulty gate
A solver model attempts the prompt; a grader from a different model family measures how much of the checklist it covers. Items a model finds too easy are returned, so every published item poses a genuine challenge to frontier models.
Lead review
With the automated gates passed, a domain lead reviews the prompt and its checklist for realism, clarity, and a fair grading key, then approves the item to proceed. Authoring is deliberately kept separate from judging.
Blind generation
6 frontier models each answer the prompt. Responses are stripped of model identity and shuffled into blind slots, so evaluators never know which lab produced which answer.
Dual expert evaluation
Two independent, domain-matched experts each grade every response: five rubric dimensions (1–5), checklist hit / partial / miss, a forced ranking of all answers, and a written rationale. Every item is judged twice, giving 2,400 expert scores.
Adjudication & gold
A senior data reviewer calibrates scoring, adjudicates disagreements between the two experts, and signs off the final, versioned gold set that ships.