Ranking run · 2026-05-20

Ranking run 2026-05-20

Claude Opus 4.7 leads with 76/100 across all domains. Tax law: Claude Opus 4.7 (79) · Medicine: Claude Opus 4.7 (80) · Law: Claude Opus 4.7 (77) · Business law: GPT-5 (78).

Status: development run — not in the feed, not on the homepage (roster test / interim run)
Run: run-20260520T074328-4a549c
Started: 2026-05-20 07:43 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 5
Judges: GPT-5, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.3
Pipeline cost: 76.74 USD
Judge agreement: 68 %

Overall across domains

#	Model	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 4.7	21 / 31	76	73–79	-3	55	1,60	—
2	GPT-5	30 / 47	74	70–78	-1	67	0,70	—
3	Gemini 2.5 Pro	22 / 26	52	48–57	-2	80	3,44	—
4	Mistral Large 2	12 / 68	49	45–54	0	49	4,28	—
5	Grok 4.3	13 / 19	48	44–51	+2	74	4,59	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 2,1 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 1,8 per answer. Relative to answer length: 1,92 per 1000 output tokens, lowest GPT-5 at 1,01.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	79	72–86	+6	0,4	2,1	2,5	100 %	1,92	77 %
2	GPT-5	54	40–67	-6	1,3	1,2	2,5	93 %	1,01	65 %
3	Gemini 2.5 Pro	30	21–40	-10	1,4	0,5	1,8	87 %	8,36	59 %
4	Grok 4.3	30	22–37	+8	1,5	0,8	2,3	93 %	9,15	56 %
5	Mistral Large 2	27	19–37	+3	1,8	1,2	3,0	100 %	6,32	55 %

Medicine · 20 Items

Ranks 1–3 statistically tied (95 % CI overlaps rank 1)

In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,6 times per answer and adds avg 0,9 claims the judge panel flags as false or unsupported; 95 % of answers carry at least one flag. The lowest value is Grok 4.3 at avg 0,4 per answer. Relative to answer length: 1,50 per 1000 output tokens, lowest GPT-5 at 0,91.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	80	73–87	-7	0,6	0,9	1,5	95 %	1,50	82 %
2	GPT-5	79	69–88	-3	0,6	0,7	1,2	85 %	0,91	81 %
3	Mistral Large 2	67	59–74	+3	0,4	0,9	1,3	95 %	2,56	68 %
4	Gemini 2.5 Pro	56	47–64	0	0,2	0,3	0,5	50 %	2,35	73 %
5	Grok 4.3	55	48–61	+4	0,3	0,2	0,4	70 %	2,06	70 %

Law · 35 Items

Ranks 1–2 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,9 claims the judge panel flags as false or unsupported; 86 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 0,2 per answer. Relative to answer length: 1,46 per 1000 output tokens, lowest GPT-5 at 0,60.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	77	70–83	-6	0,3	0,9	1,2	86 %	1,46	83 %
2	GPT-5	77	71–82	+1	0,2	0,8	1,0	83 %	0,60	73 %
3	Gemini 2.5 Pro	62	56–66	-1	0,2	0,1	0,2	29 %	1,31	72 %
4	Grok 4.3	50	45–56	-4	0,3	0,2	0,4	51 %	3,41	68 %
5	Mistral Large 2	46	38–54	-4	1,1	0,7	1,8	83 %	6,12	62 %

Business law · 30 Items

Ranks 1–2 statistically tied (95 % CI overlaps rank 1)

In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 1,0 claims the judge panel flags as false or unsupported; 93 % of answers carry at least one flag. The lowest value is Gemini 2.5 Pro at avg 1,1 per answer. Relative to answer length: 0,53 per 1000 output tokens.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5	78	72–83	+1	0,3	1,0	1,2	93 %	0,53	73 %
2	Claude Opus 4.7	72	67–76	-1	0,5	1,7	2,3	97 %	1,60	66 %
3	Mistral Large 2	52	46–59	0	0,7	1,5	2,2	93 %	3,40	56 %
4	Gemini 2.5 Pro	50	43–56	+2	0,6	0,6	1,1	83 %	3,77	64 %
5	Grok 4.3	49	42–56	+4	0,7	0,7	1,4	77 %	4,43	60 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).