Ranking run 2026-05-12
Claude Opus 4.7 leads with 77/100 across all domains. Tax law: Claude Opus 4.7 (71) · Medicine: Claude Opus 4.7 (85) · Law: Claude Opus 4.7 (80) · Business law: GPT-5 (77).
Overall across domains
| # | Model | top100 | 95 % CI | Δ prev. | failures | sec median/p90 | hallu band | per 1000 tok. | Δ own family |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 0 | 17 / 27 | 77 | 72–81 | +2 | 51 | 1,56 | — |
| 2 | GPT-5 | 0 | 29 / 47 | 73 | 65–79 | +3 | 59 | 0,72 | — |
| 3 | Gemini 2.5 Pro | 0 | 23 / 28 | 48 | 42–54 | +1 | 71 | 4,42 | — |
| 3 | Mistral Large 2 | 0 | 15 / 84 | 48 | 39–56 | +5 | 45 | 4,00 | — |
Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).
Tax law · 8 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,6 times per answer and adds avg 2,0 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 1,9 per answer. Relative to answer length: 2,07 per 1000 output tokens, lowest GPT-5 at 0,80.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 71 | 64–79 | +2 | 0,6 | 2,0 | 2,6 | 100 % | 2,07 | 82 % |
| 2 | GPT-5 | 63 | 49–76 | +12 | 0,9 | 1,5 | 2,3 | 100 % | 0,80 | 77 % |
| 3 | Gemini 2.5 Pro | 32 | 20–46 | +1 | 1,3 | 0,7 | 1,9 | 88 % | 7,79 | 65 % |
| 4 | Mistral Large 2 | 12 | 7–17 | +1 | 2,5 | 1,1 | 3,5 | 100 % | 7,74 | 60 % |
Medicine · 9 Items
Ranks 1–3 statistically tied (95 % CI overlaps rank 1)
In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,6 per answer. Relative to answer length: 0,84 per 1000 output tokens, lowest GPT-5 at 0,76.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 85 | 75–93 | +3 | 0,2 | 0,7 | 0,9 | 100 % | 0,84 | 84 % |
| 2 | GPT-5 | 85 | 78–92 | +1 | 0,3 | 0,7 | 1,1 | 89 % | 0,76 | 80 % |
| 3 | Mistral Large 2 | 70 | 64–77 | +3 | 0,2 | 1,2 | 1,4 | 100 % | 2,22 | 80 % |
| 4 | Gemini 2.5 Pro | 54 | 44–63 | +1 | 0,4 | 0,3 | 0,6 | 67 % | 2,43 | 70 % |
Law · 8 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,1 times per answer and adds avg 0,9 claims the judge panel flags as false or unsupported; 88 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,4 per answer. Relative to answer length: 1,12 per 1000 output tokens, lowest GPT-5 at 0,68.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 80 | 75–85 | +1 | 0,1 | 0,9 | 1,0 | 88 % | 1,12 | 87 % |
| 2 | GPT-5 | 64 | 44–77 | -7 | 0,9 | 0,5 | 1,3 | 75 % | 0,68 | 70 % |
| 3 | Mistral Large 2 | 56 | 48–65 | +9 | 0,4 | 0,5 | 0,9 | 75 % | 2,95 | 60 % |
| 4 | Gemini 2.5 Pro | 51 | 41–58 | -4 | 0,4 | 0,1 | 0,4 | 38 % | 1,97 | 64 % |
Business law · 8 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,6 times per answer and adds avg 1,1 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. Relative to answer length: 0,64 per 1000 output tokens.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5 | 77 | 68–84 | +3 | 0,6 | 1,1 | 1,8 | 100 % | 0,64 | 71 % |
| 2 | Claude Opus 4.7 | 71 | 63–81 | +2 | 0,6 | 2,8 | 3,4 | 100 % | 1,89 | 60 % |
| 3 | Gemini 2.5 Pro | 53 | 38–65 | +7 | 0,9 | 0,9 | 1,8 | 100 % | 4,92 | 55 % |
| 4 | Mistral Large 2 | 49 | 39–59 | +4 | 1,1 | 1,9 | 2,9 | 100 % | 3,85 | 51 % |
Raw data of this run
raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).