Ranking run 2026-05-28
Claude Opus 4.7 leads with 79/100 across all domains. Tax law: Claude Opus 4.7 (76) · Medicine: Claude Opus 4.7 (87) · Law: Claude Opus 4.7 (82) · Business law: GPT-5 (78).
Overall across domains
| # | Model | top100 | 95 % CI | Δ prev. | failures | sec median/p90 | hallu band | per 1000 tok. | Δ own family |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 0 | 20 / 27 | 79 | 77–82 | +3 | 57 | 1,53 | — |
| 2 | GPT-5 | 0 | 39 / 64 | 75 | 71–78 | +1 | 68 | 0,64 | — |
| 3 | Gemini 2.5 Pro | 0 | 20 / 25 | 52 | 48–56 | 0 | 77 | 4,03 | — |
| 4 | Mistral Large 2 | 0 | 10 / 69 | 50 | 46–55 | +1 | 49 | 4,24 | — |
| 5 | Grok 4.3 | 0 | 8 / 16 | 47 | 43–51 | -1 | 77 | 4,53 | — |
Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).
Tax law · 15 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 1,9 claims the judge panel flags as false or unsupported; 93 % of answers carry at least one flag. The lowest value is Grok 4.3 at avg 1,5 per answer. Relative to answer length: 1,94 per 1000 output tokens, lowest GPT-5 at 0,73.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 76 | 69–83 | -3 | 0,5 | 1,9 | 2,4 | 93 % | 1,94 | 81 % |
| 2 | GPT-5 | 61 | 51–72 | +7 | 0,8 | 1,1 | 1,9 | 100 % | 0,73 | 63 % |
| 3 | Gemini 2.5 Pro | 33 | 24–44 | +3 | 1,2 | 0,6 | 1,7 | 87 % | 7,62 | 65 % |
| 4 | Grok 4.3 | 33 | 24–41 | +3 | 1,1 | 0,5 | 1,5 | 80 % | 6,56 | 61 % |
| 5 | Mistral Large 2 | 26 | 17–35 | -1 | 1,9 | 1,4 | 3,3 | 100 % | 7,01 | 59 % |
Medicine · 20 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 1,1 claims the judge panel flags as false or unsupported; 90 % of answers carry at least one flag. The lowest value is Grok 4.3 at avg 0,4 per answer. Relative to answer length: 1,36 per 1000 output tokens, lowest GPT-5 at 0,81.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 87 | 82–91 | +7 | 0,4 | 1,1 | 1,4 | 90 % | 1,36 | 85 % |
| 2 | GPT-5 | 79 | 71–86 | 0 | 0,5 | 0,6 | 1,1 | 90 % | 0,81 | 77 % |
| 3 | Mistral Large 2 | 66 | 59–73 | -1 | 0,4 | 0,9 | 1,2 | 95 % | 2,24 | 71 % |
| 4 | Gemini 2.5 Pro | 56 | 48–66 | 0 | 0,3 | 0,3 | 0,5 | 55 % | 2,39 | 75 % |
| 5 | Grok 4.3 | 55 | 47–62 | 0 | 0,3 | 0,2 | 0,4 | 50 % | 2,24 | 73 % |
Law · 35 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 71 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,5 per answer. Relative to answer length: 0,99 per 1000 output tokens, lowest GPT-5 at 0,50.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 82 | 79–86 | +5 | 0,2 | 0,7 | 0,9 | 71 % | 0,99 | 83 % |
| 2 | GPT-5 | 74 | 68–79 | -3 | 0,2 | 0,7 | 0,8 | 71 % | 0,50 | 70 % |
| 3 | Gemini 2.5 Pro | 61 | 55–66 | -1 | 0,3 | 0,2 | 0,5 | 37 % | 2,34 | 73 % |
| 4 | Grok 4.3 | 48 | 42–54 | -2 | 0,4 | 0,2 | 0,5 | 49 % | 4,62 | 68 % |
| 5 | Mistral Large 2 | 48 | 41–56 | +2 | 1,0 | 0,6 | 1,6 | 80 % | 5,15 | 66 % |
Business law · 30 Items
Ranks 1–2 statistically tied (95 % CI overlaps rank 1)
In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 1,2 claims the judge panel flags as false or unsupported; 93 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 1,3 per answer. Relative to answer length: 0,65 per 1000 output tokens.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5 | 78 | 72–84 | 0 | 0,4 | 1,2 | 1,5 | 93 % | 0,65 | 72 % |
| 2 | Claude Opus 4.7 | 72 | 67–78 | 0 | 0,6 | 2,0 | 2,6 | 100 % | 1,83 | 68 % |
| 3 | Mistral Large 2 | 54 | 46–61 | +2 | 0,7 | 1,7 | 2,4 | 97 % | 3,83 | 53 % |
| 4 | Gemini 2.5 Pro | 48 | 41–56 | -2 | 0,7 | 0,6 | 1,3 | 77 % | 4,80 | 62 % |
| 5 | Grok 4.3 | 47 | 41–54 | -2 | 0,6 | 0,7 | 1,3 | 83 % | 4,66 | 55 % |
Raw data of this run
raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).