AI-Roundtable Leaderboard
Ranking run · 2026-06-18

Ranking run 2026-06-18

Claude Opus 4.7 leads with 80/100 across all domains. Tax law: Claude Opus 4.7 (70) · Medicine: Claude Opus 4.7 (85) · Law: Claude Opus 4.7 (84) · Business law: GPT-5 (80).

Status
development run — not in the feed, not on the homepage (roster test / interim run)
Run
run-20260618T165158-d2543a
Started
2026-06-18 16:51 UTC
Items
100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models
7
Judges
GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank
2026-05-17-v1
Code version
0.2.6
Pipeline cost
87.07 USD
Judge agreement
68 %

Overall across domains

#Modeltop10095 % CIΔ prev.failuressec median/p90hallu bandper 1000 tok.Δ own family
1Claude Opus 4.7019 / 298077–83+1601,49—
2GPT-5024 / 387672–79+2690,63—
3Claude Opus 4.8018 / 287470–780691,02—
4Kimi K2.60186 / 7725651–61—450,30—
5Gemini 2.5 Pro021 / 265349–57-1793,66—
6Mistral Large 2010 / 705145–55+2474,60—
7Grok 4.307 / 84439–49-4686,30—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–3 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,7 times per answer and adds avg 1,8 claims the judge panel flags as false or unsupported; 93 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 2,0 per answer. Relative to answer length: 2,06 per 1000 output tokens, lowest Kimi K2.6 at 0,37.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1Claude Opus 4.77064–77-50,71,82,593 %2,0671 %
2Claude Opus 4.86963–74+30,81,22,0100 %1,4276 %
3GPT-56249–73+31,11,12,2100 %0,8565 %
4Kimi K2.63726–48—1,51,83,3100 %0,3760 %
5Gemini 2.5 Pro3525–46-31,30,72,087 %9,1860 %
6Mistral Large 22818–38-21,71,33,0100 %6,4460 %
7Grok 4.32314–32-31,70,72,4100 %10,3651 %

Medicine · 20 Items

Ranks 1–3 statistically tied (95 % CI overlaps rank 1)

In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 95 % of answers carry at least one flag. The lowest value is Grok 4.3 at avg 0,4 per answer. Relative to answer length: 1,12 per 1000 output tokens, lowest Kimi K2.6 at 0,18.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1Claude Opus 4.78579–9000,40,81,195 %1,1288 %
2GPT-58477–89+40,50,61,090 %0,7778 %
3Claude Opus 4.87971–8600,30,30,675 %0,5676 %
4Mistral Large 26861–74-10,40,81,295 %2,2875 %
5Kimi K2.66659–74—0,50,51,095 %0,1862 %
6Gemini 2.5 Pro5952–66-20,20,40,555 %2,3174 %
7Grok 4.35346–60+10,30,10,440 %1,7968 %

Law · 35 Items

Ranks 1–3 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 77 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,3 per answer. Relative to answer length: 1,10 per 1000 output tokens, lowest Kimi K2.6 at 0,26.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1Claude Opus 4.78480–88+40,20,80,977 %1,1084 %
2Claude Opus 4.87872–83-20,30,50,871 %0,8877 %
3GPT-57466–80+10,40,61,089 %0,5676 %
4Gemini 2.5 Pro6155–67+20,20,10,329 %1,4074 %
5Kimi K2.65749–65—1,00,91,883 %0,2668 %
6Mistral Large 24436–52+21,30,92,183 %7,0760 %
7Grok 4.34234–49-91,10,31,266 %10,7068 %

Business law · 30 Items

Ranks 1–2 statistically tied (95 % CI overlaps rank 1)

In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 97 % of answers carry at least one flag. Relative to answer length: 0,51 per 1000 output tokens, lowest Kimi K2.6 at 0,37.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1GPT-58075–8500,50,71,297 %0,5164 %
2Claude Opus 4.77873–83+10,41,92,3100 %1,6970 %
3Claude Opus 4.86963–75+10,61,21,8100 %1,1370 %
4Kimi K2.65850–66—0,82,02,9100 %0,3753 %
5Mistral Large 25851–64+40,81,52,393 %3,7655 %
6Grok 4.35143–59-20,80,61,477 %4,6756 %
7Gemini 2.5 Pro4841–55-40,70,51,287 %4,0755 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).