Ranking run 2026-05-14
Claude Opus 4.7 leads with 75/100 across all domains. Tax law: Claude Opus 4.7 (68) · Medicine: Claude Opus 4.7 (82) · Law: Claude Opus 4.7 (77) · Business law: GPT-5 (80).
Overall across domains
| # | Model | top100 | 95 % CI | Δ prev. | failures | sec median/p90 | hallu band | per 1000 tok. | Δ own family |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 0 | 20 / 31 | 75 | 72–78 | -2 | 49 | 1,59 | — |
| 2 | GPT-5 | 0 | 32 / 56 | 70 | 66–74 | +1 | 62 | 0,68 | — |
| 3 | Mistral Large 2 | 0 | 16 / 79 | 46 | 41–51 | -1 | 38 | 4,36 | — |
| 4 | Gemini 2.5 Pro | 0 | 23 / 27 | 45 | 41–48 | -2 | 72 | 4,25 | — |
Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).
Tax law · 24 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,8 times per answer and adds avg 2,1 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 1,8 per answer. Relative to answer length: 2,09 per 1000 output tokens, lowest GPT-5 at 0,87.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 68 | 62–74 | 0 | 0,8 | 2,1 | 2,8 | 100 % | 2,09 | 78 % |
| 2 | GPT-5 | 53 | 44–62 | -3 | 1,1 | 1,3 | 2,4 | 96 % | 0,87 | 72 % |
| 3 | Gemini 2.5 Pro | 30 | 23–37 | +5 | 1,4 | 0,5 | 1,8 | 88 % | 7,81 | 59 % |
| 4 | Mistral Large 2 | 13 | 9–17 | -4 | 2,2 | 1,5 | 3,7 | 100 % | 7,62 | 62 % |
Medicine · 27 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,9 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,4 per answer. Relative to answer length: 1,15 per 1000 output tokens, lowest GPT-5 at 0,69.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 82 | 76–87 | -3 | 0,3 | 0,9 | 1,2 | 100 % | 1,15 | 83 % |
| 2 | GPT-5 | 80 | 75–84 | -6 | 0,4 | 0,6 | 1,0 | 96 % | 0,69 | 68 % |
| 3 | Mistral Large 2 | 67 | 62–72 | -2 | 0,6 | 0,9 | 1,5 | 96 % | 2,44 | 69 % |
| 4 | Gemini 2.5 Pro | 54 | 49–57 | -2 | 0,3 | 0,2 | 0,4 | 74 % | 1,92 | 71 % |
Law · 24 Items
In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 83 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,5 per answer. Relative to answer length: 1,35 per 1000 output tokens, lowest GPT-5 at 0,54.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 77 | 71–82 | -1 | 0,4 | 0,8 | 1,2 | 83 % | 1,35 | 73 % |
| 2 | GPT-5 | 66 | 61–70 | +6 | 0,3 | 0,6 | 1,0 | 71 % | 0,54 | 64 % |
| 3 | Gemini 2.5 Pro | 51 | 45–57 | -4 | 0,3 | 0,2 | 0,5 | 46 % | 2,38 | 70 % |
| 4 | Mistral Large 2 | 49 | 45–54 | +1 | 0,5 | 0,8 | 1,2 | 88 % | 3,78 | 59 % |
Business law · 24 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 1,1 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. Relative to answer length: 0,57 per 1000 output tokens.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5 | 80 | 76–84 | +5 | 0,5 | 1,1 | 1,6 | 100 % | 0,57 | 74 % |
| 2 | Claude Opus 4.7 | 72 | 65–78 | -3 | 0,6 | 2,3 | 2,9 | 100 % | 1,62 | 71 % |
| 3 | Mistral Large 2 | 51 | 43–58 | -1 | 1,5 | 2,0 | 3,5 | 100 % | 4,32 | 58 % |
| 4 | Gemini 2.5 Pro | 44 | 37–51 | -6 | 1,2 | 0,6 | 1,7 | 92 % | 4,69 | 44 % |
Raw data of this run
raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).