Ranking run 2026-05-10
Claude Opus 4.7 leads with 77/100 across all domains. Tax law: Claude Opus 4.7 (66) · Medicine: Claude Opus 4.7 (86) · Law: Claude Opus 4.7 (78) · Business law: Claude Opus 4.7 (78).
Overall across domains
| # | Model | top100 | 95 % CI | Δ prev. | failures | sec median/p90 | hallu band | per 1000 tok. | Δ own family |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5 n=16/33 | 17 | 17 / 24 | 78 | 72–84 | — | 77 | 0,65 | — |
| 2 | Claude Opus 4.7 | 0 | 20 / 34 | 77 | 72–82 | — | 49 | 1,61 | — |
| 3 | Mistral Large 2 | 0 | 15 / 79 | 46 | 36–55 | — | 41 | 4,40 | — |
| 4 | Gemini 2.5 Pro n=9/33 | 24 | 16 / 33 | 32 | 19–47 | — | 98 | 0,73 | — |
Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).
Tax law · 8 Items
In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 1,4 times per answer and adds avg 1,5 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,0 per answer. Relative to answer length: 2,16 per 1000 output tokens, lowest Gemini 2.5 Pro at 0,00.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 66 | 59–73 | — | 1,4 | 1,5 | 2,9 | 100 % | 2,16 | 77 % |
| 2 | Mistral Large 2 | 9 | 3–15 | — | 2,5 | 1,7 | 4,2 | 100 % | 8,43 | 64 % |
| 3 | Gemini 2.5 Pro | 0 | 0–0 | — | 0,0 | 0,0 | 0,0 | 0 % | 0,00 | 100 % |
Medicine · 9 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 1,0 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,2 per answer. Relative to answer length: 1,43 per 1000 output tokens, lowest GPT-5 at 0,86.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 86 | 78–93 | — | 0,3 | 1,0 | 1,4 | 100 % | 1,43 | 90 % |
| 2 | GPT-5 | 84 | 78–90 | — | 0,4 | 0,7 | 1,1 | 89 % | 0,86 | 79 % |
| 3 | Mistral Large 2 | 68 | 58–77 | — | 0,2 | 1,0 | 1,2 | 89 % | 2,00 | 71 % |
| 4 | Gemini 2.5 Pro | 43 | 27–58 | — | 0,0 | 0,2 | 0,2 | 50 % | 1,28 | 78 % |
Law · 8 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 88 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,0 per answer. Relative to answer length: 1,29 per 1000 output tokens, lowest Gemini 2.5 Pro at 0,00.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 78 | 70–85 | — | 0,3 | 0,8 | 1,1 | 88 % | 1,29 | 79 % |
| 2 | GPT-5 | 70 | 63–78 | — | 0,1 | 0,5 | 0,6 | 86 % | 0,42 | 61 % |
| 3 | Mistral Large 2 | 53 | 49–58 | — | 0,5 | 0,8 | 1,3 | 88 % | 3,98 | 58 % |
| 4 | Gemini 2.5 Pro | 38 | 11–59 | — | 0,0 | 0,0 | 0,0 | 0 % | 0,00 | 79 % |
Business law · 8 Items
In Business law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 2,3 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,0 per answer. Relative to answer length: 1,46 per 1000 output tokens, lowest Gemini 2.5 Pro at 0,00.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 78 | 68–88 | — | 0,5 | 2,3 | 2,8 | 100 % | 1,46 | 70 % |
| 2 | Mistral Large 2 | 50 | 34–65 | — | 1,3 | 1,6 | 2,9 | 100 % | 4,03 | 56 % |
| 3 | Gemini 2.5 Pro | 7 | 7–7 | — | 0,0 | 0,0 | 0,0 | 0 % | 0,00 | 90 % |
Raw data of this run
raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).