AI-Roundtable Leaderboard
Ranking run · 2026-09-07

Ranking run 2026-09-07

Claude Opus 5 leads with 86/100 across all domains. Tax law: Claude Opus 5 (82) · Medicine: Claude Fable 5.1 (90) · Law: Claude Opus 5 (90) · Business law: Claude Opus 5 (81).

Status
official run (feed, archive, homepage)
Run
run-20260907T181839-5742df
Started
2026-09-07 18:18 UTC
Items
100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models
14
Judges
GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank
2026-05-17-v1
Code version
0.2.8
Pipeline cost
233.07 USD
Judge agreement
72 %

Overall across domains

#Modeltop10095 % CIΔ prev.failuressec median/p90hallu bandper 1000 tok.Δ own family
1Claude Opus 5064 / 1198682–89—410,44+3,1
2Claude Fable 5.1 n=98/1002 (1 T)48 / 868278–85—430,59+2,1
3Claude Opus 4.7019 / 298077–83-1551,58+1,7
4GLM-5.3 n=60/10040 (1 T)110 / 3777974–83—520,240,0
5Kimi K3 n=83/10017 (17 T)150 / 2967872–82-2420,380,0
6GPT-5029 / 487773–81+4670,66-2,5
7GPT-5.6 Sol0100 / 2237673–80-1630,32-3,8
8Claude Opus 4.8019 / 297572–780691,02+2,8
8GPT-6 Astra097 / 2137571–78—640,35-3,8
9Grok 4.5030 / 516663–69+7832,050,0
10Grok 4.6065 / 1126359–67—743,090,0
11DeepSeek V4 Pro n=98/100277 / 1946056–64—680,190,0
12Gemini 3.1 Pro022 / 325652–600754,050,0
13Mistral Large 208 / 155045–55-1464,58+4,1

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–8 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 5 (top score) contradicts a stored key fact on average 1,0 times per answer and adds avg 2,4 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,2 per answer. Relative to answer length: 0,43 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,26.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1Claude Opus 58273–89—1,02,43,4100 %0,4375 %
2Claude Fable 5.17566–84—1,12,73,9100 %0,6169 %
3Claude Opus 4.77366–80-20,61,82,4100 %1,8381 %
4Claude Opus 4.86964–74+30,41,11,593 %1,1377 %
5GPT-6 Astra6960–77—1,01,52,5100 %0,4071 %
6GPT-5.6 Sol6856–79-31,31,22,5100 %0,3173 %
7Kimi K36441–79+91,12,53,6100 %0,4176 %
8GPT-56249–75-11,01,22,293 %0,8268 %
9Grok 4.66050–69—1,00,91,893 %4,1470 %
10Grok 4.55547–63+30,80,51,273 %3,5364 %
11DeepSeek V4 Pro5042–58—0,91,52,479 %0,2662 %
12GLM-5.34618–74—2,32,64,7100 %0,3653 %
13Gemini 3.1 Pro4538–53-21,00,51,473 %5,8164 %
14Mistral Large 22918–4101,81,63,4100 %7,0156 %

Medicine · 19 Items

Ranks 1–9 statistically tied (95 % CI overlaps rank 1)

In Medicine, Claude Fable 5.1 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,9 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,2 per answer. Relative to answer length: 0,52 per 1000 output tokens, lowest GLM-5.3 at 0,20.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1Claude Fable 5.19085–95—0,30,91,2100 %0,5284 %
2Claude Opus 59083–95—0,51,21,7100 %0,4887 %
3Kimi K38780–94+50,51,01,4100 %0,4080 %
4Claude Opus 4.78579–91-20,50,61,095 %1,0184 %
5GPT-58579–90+50,40,50,990 %0,6479 %
6GLM-5.38480–89—0,40,71,1100 %0,2073 %
7GPT-6 Astra8479–89—0,50,50,995 %0,4682 %
8GPT-5.6 Sol8378–88-10,40,60,9100 %0,6377 %
9Claude Opus 4.88175–87+20,20,60,895 %0,7571 %
10DeepSeek V4 Pro7061–77—0,40,30,670 %0,2576 %
11Grok 4.57064–75+70,10,20,250 %0,8070 %
12Mistral Large 26863–74-30,50,81,395 %2,4062 %
13Grok 4.66254–70—0,30,30,565 %2,2468 %
14Gemini 3.1 Pro6053–67-60,30,30,570 %2,2775 %

Law · 35 Items

Ranks 1–6 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 5 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 1,4 claims the judge panel flags as false or unsupported; 94 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,4 per answer. Relative to answer length: 0,47 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,13.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1Claude Opus 59084–94—0,31,41,794 %0,4778 %
2Claude Fable 5.18783–91—0,41,11,594 %0,5380 %
3Claude Opus 4.78175–86-30,31,01,383 %1,4878 %
4Claude Opus 4.88177–8500,20,50,774 %0,7980 %
5GLM-5.37973–85—0,71,21,989 %0,2376 %
6Kimi K37870–84-40,71,21,994 %0,3480 %
7GPT-57569–80+30,40,71,191 %0,6373 %
8GPT-5.6 Sol7569–81+20,50,81,289 %0,3472 %
9GPT-6 Astra7165–77—0,70,71,391 %0,4166 %
10Grok 4.56862–73+120,40,10,449 %2,1272 %
11Gemini 3.1 Pro6356–69+40,40,30,757 %3,4571 %
12Grok 4.66356–70—0,60,30,863 %3,7470 %
13DeepSeek V4 Pro5952–66—0,60,40,966 %0,1368 %
14Mistral Large 24537–53+21,30,92,183 %7,1868 %

Business law · 30 Items

Ranks 1–8 statistically tied (95 % CI overlaps rank 1)

In Business law, Claude Opus 5 (top score) contradicts a stored key fact on average 0,8 times per answer and adds avg 2,2 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,0 per answer. Relative to answer length: 0,41 per 1000 output tokens, lowest GLM-5.3 at 0,21.

#Modeltop10095 % CIΔ prev.contradicts key factextra false/unsupp.avg totalanswers affectedper 1000 tok.IRR
1Claude Opus 58174–87—0,82,23,0100 %0,4176 %
2GPT-58175–87+60,40,91,497 %0,6079 %
3Claude Opus 4.77975–83+30,32,32,6100 %1,8071 %
4GLM-5.37965–89—0,52,12,6100 %0,2177 %
5GPT-5.6 Sol7670–83-30,61,01,6100 %0,2674 %
6GPT-6 Astra7669–82—0,60,91,497 %0,2573 %
7Claude Fable 5.17363–81—0,82,23,1100 %0,6468 %
8Kimi K37362–82-60,72,63,3100 %0,3964 %
9Grok 4.56863–73+50,50,51,080 %1,9563 %
10Claude Opus 4.86662–71-50,81,21,997 %1,2763 %
11Grok 4.66456–71—0,70,71,393 %2,5968 %
12DeepSeek V4 Pro5850–65—0,70,91,597 %0,2256 %
13Mistral Large 25547–64-40,81,32,1100 %3,4256 %
14Gemini 3.1 Pro5143–59-10,90,61,490 %4,6361 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).