Ranking run 2026-05-29
Claude Opus 4.7 leads with 79/100 across all domains. Tax law: Claude Opus 4.7 (71) · Medicine: Claude Opus 4.7 (84) · Law: Claude Opus 4.7 (82) · Business law: GPT-5 (80).
Overall across domains
| # | Model | top100 | 95 % CI | Δ prev. | failures | sec median/p90 | hallu band | per 1000 tok. | Δ own family |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 0 | 19 / 31 | 79 | 77–82 | — | 57 | 1,54 | — |
| 2 | GPT-5 | 0 | 34 / 59 | 75 | 71–79 | — | 67 | 0,67 | — |
| 3 | Claude Opus 4.8 | 0 | 18 / 28 | 74 | 71–77 | — | 72 | 0,91 | — |
| 4 | Gemini 2.5 Pro | 0 | 21 / 26 | 55 | 51–59 | — | 80 | 3,45 | — |
| 5 | Mistral Large 2 | 0 | 9 / 65 | 51 | 47–57 | — | 47 | 4,62 | — |
| 6 | Grok 4.3 | 0 | 6 / 10 | 47 | 43–51 | — | 76 | 4,80 | — |
Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).
Tax law · 15 Items
Ranks 1–3 statistically tied (95 % CI overlaps rank 1)
In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 2,0 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Claude Opus 4.8 at avg 1,4 per answer. Relative to answer length: 1,95 per 1000 output tokens, lowest GPT-5 at 0,83.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 71 | 66–77 | — | 0,5 | 2,0 | 2,4 | 100 % | 1,95 | 78 % |
| 2 | Claude Opus 4.8 | 69 | 63–74 | — | 0,6 | 0,9 | 1,4 | 100 % | 1,02 | 79 % |
| 3 | GPT-5 | 63 | 52–73 | — | 0,9 | 1,3 | 2,2 | 100 % | 0,83 | 69 % |
| 4 | Gemini 2.5 Pro | 38 | 28–48 | — | 1,2 | 0,6 | 1,8 | 80 % | 8,03 | 60 % |
| 5 | Mistral Large 2 | 28 | 19–39 | — | 1,9 | 1,6 | 3,4 | 93 % | 7,55 | 50 % |
| 6 | Grok 4.3 | 27 | 18–37 | — | 1,5 | 1,0 | 2,4 | 100 % | 10,98 | 60 % |
Medicine · 20 Items
Ranks 1–3 statistically tied (95 % CI overlaps rank 1)
In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,4 per answer. Relative to answer length: 1,15 per 1000 output tokens, lowest Claude Opus 4.8 at 0,68.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 84 | 77–90 | — | 0,5 | 0,8 | 1,1 | 100 % | 1,15 | 81 % |
| 2 | GPT-5 | 81 | 72–89 | — | 0,6 | 0,5 | 1,0 | 80 % | 0,75 | 78 % |
| 3 | Claude Opus 4.8 | 79 | 72–84 | — | 0,4 | 0,4 | 0,7 | 80 % | 0,68 | 72 % |
| 4 | Mistral Large 2 | 67 | 60–74 | — | 0,4 | 0,9 | 1,2 | 95 % | 2,47 | 70 % |
| 5 | Gemini 2.5 Pro | 61 | 52–67 | — | 0,1 | 0,3 | 0,4 | 60 % | 1,71 | 73 % |
| 6 | Grok 4.3 | 55 | 47–64 | — | 0,3 | 0,2 | 0,4 | 60 % | 2,18 | 75 % |
Law · 35 Items
Ranks 1–3 statistically tied (95 % CI overlaps rank 1)
In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 86 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,2 per answer. Relative to answer length: 1,22 per 1000 output tokens, lowest GPT-5 at 0,58.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 | 82 | 77–86 | — | 0,2 | 0,8 | 1,0 | 86 % | 1,22 | 82 % |
| 2 | Claude Opus 4.8 | 79 | 74–84 | — | 0,2 | 0,4 | 0,6 | 66 % | 0,61 | 76 % |
| 3 | GPT-5 | 73 | 66–80 | — | 0,4 | 0,6 | 1,0 | 80 % | 0,58 | 70 % |
| 4 | Gemini 2.5 Pro | 62 | 58–67 | — | 0,1 | 0,1 | 0,2 | 34 % | 1,21 | 68 % |
| 5 | Grok 4.3 | 48 | 42–54 | — | 0,3 | 0,2 | 0,4 | 34 % | 3,60 | 67 % |
| 6 | Mistral Large 2 | 46 | 38–54 | — | 1,3 | 0,7 | 1,9 | 77 % | 6,53 | 64 % |
Business law · 30 Items
Ranks 1–3 statistically tied (95 % CI overlaps rank 1)
In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 1,0 claims the judge panel flags as false or unsupported; 97 % of answers carry at least one flag. The lowest value is Grok 4.3 at avg 1,1 per answer. Relative to answer length: 0,63 per 1000 output tokens.
| # | Model | top100 | 95 % CI | Δ prev. | contradicts key fact | extra false/unsupp. | avg total | answers affected | per 1000 tok. | IRR |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | GPT-5 | 80 | 73–86 | — | 0,4 | 1,0 | 1,4 | 97 % | 0,63 | 75 % |
| 2 | Claude Opus 4.7 | 77 | 72–81 | — | 0,3 | 2,2 | 2,5 | 100 % | 1,76 | 68 % |
| 3 | Claude Opus 4.8 | 68 | 63–74 | — | 0,5 | 1,4 | 1,9 | 97 % | 1,18 | 62 % |
| 4 | Mistral Large 2 | 59 | 52–65 | — | 0,7 | 1,6 | 2,3 | 97 % | 3,62 | 58 % |
| 5 | Gemini 2.5 Pro | 52 | 45–58 | — | 0,7 | 0,6 | 1,3 | 90 % | 4,29 | 61 % |
| 6 | Grok 4.3 | 50 | 43–57 | — | 0,6 | 0,6 | 1,1 | 87 % | 4,12 | 54 % |
Raw data of this run
raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).