Ranking run · 2026-06-20

Ranking run 2026-06-20

Claude Opus 4.7 leads with 80/100 across all domains. Tax law: Claude Opus 4.7 (78) · Medicine: GPT-5.5 Pro (87) · Law: Claude Opus 4.7 (82) · Business law: GPT-5.5 Pro (78).

Status: development run — not in the feed, not on the homepage (roster test / interim run)
Run: run-20260620T073912-d2125c
Started: 2026-06-20 07:39 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 8
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.6
Pipeline cost: 122.61 USD
Judge agreement: 68 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 4.7	0	20 / 29	80	77–83	+2	59	1,48	—
2	GPT-5.5 Pro n=99/100	1	199 / 831	78	75–82	-2	78	0,10	—
3	GPT-5	0	28 / 44	76	72–79	+4	70	0,60	—
4	Claude Opus 4.8	0	18 / 27	74	71–77	0	73	0,90	—
5	Kimi K2.6 n=99/100	1	122 / 266	59	54–64	+2	51	0,25	—
6	Gemini 3.1 Pro	0	23 / 32	54	50–58	0	75	4,15	—
7	Mistral Large 2	0	10 / 68	51	46–56	-1	50	4,30	—
8	Grok 4.3	0	5 / 7	48	43–52	0	73	5,19	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–3 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 1,6 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 1,1 per answer. Relative to answer length: 1,65 per 1000 output tokens, lowest GPT-5.5 Pro at 0,11.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	78	73–84	+7	0,5	1,6	2,1	100 %	1,65	80 %
2	GPT-5.5 Pro	68	60–76	+1	0,9	0,7	1,6	100 %	0,11	68 %
3	GPT-5	67	54–79	+12	1,1	1,0	2,0	100 %	0,79	74 %
4	Claude Opus 4.8	65	59–72	-2	0,6	1,0	1,6	100 %	1,15	74 %
5	Gemini 3.1 Pro	44	36–51	0	0,8	0,5	1,1	73 %	4,66	56 %
6	Kimi K2.6	30	18–42	-3	1,9	2,0	3,9	100 %	0,40	46 %
7	Grok 4.3	28	21–37	+1	1,1	0,7	1,9	93 %	8,55	54 %
8	Mistral Large 2	27	18–37	+2	1,9	1,6	3,5	100 %	7,21	52 %

Medicine · 20 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Medicine, GPT-5.5 Pro (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 0,3 claims the judge panel flags as false or unsupported; 80 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 0,4 per answer. Relative to answer length: 0,19 per 1000 output tokens, lowest Kimi K2.6 at 0,17.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5.5 Pro	87	83–91	+1	0,5	0,3	0,7	80 %	0,19	81 %
2	Claude Opus 4.7	83	75–90	0	0,4	0,9	1,3	100 %	1,26	88 %
3	Claude Opus 4.8	81	75–87	-1	0,3	0,4	0,6	90 %	0,59	70 %
4	GPT-5	78	70–86	-2	0,6	0,5	1,0	100 %	0,73	70 %
5	Kimi K2.6	70	63–77	+6	0,4	0,6	0,9	90 %	0,17	67 %
6	Mistral Large 2	69	63–75	-2	0,5	0,5	1,0	85 %	1,89	72 %
7	Gemini 3.1 Pro	63	57–70	+3	0,3	0,2	0,4	60 %	2,00	70 %
8	Grok 4.3	60	50–68	+4	0,2	0,4	0,6	75 %	2,86	78 %

Law · 35 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 83 % of answers carry at least one flag. The lowest value is GPT-5.5 Pro at avg 0,6 per answer. Relative to answer length: 1,22 per 1000 output tokens, lowest GPT-5.5 Pro at 0,08.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	82	77–86	+3	0,3	0,7	1,0	83 %	1,22	82 %
2	Claude Opus 4.8	80	74–85	+4	0,2	0,5	0,7	71 %	0,70	76 %
3	GPT-5	77	73–81	+7	0,3	0,5	0,7	77 %	0,39	71 %
4	GPT-5.5 Pro	77	70–83	0	0,2	0,4	0,6	71 %	0,08	78 %
5	Kimi K2.6	63	57–69	+2	0,6	0,5	1,1	77 %	0,15	66 %
6	Gemini 3.1 Pro	58	51–64	-1	0,5	0,3	0,7	60 %	3,97	71 %
7	Grok 4.3	49	43–54	+1	0,5	0,2	0,6	60 %	4,97	67 %
8	Mistral Large 2	45	37–52	-4	1,2	0,7	1,8	80 %	6,10	62 %

Business law · 30 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Business law, GPT-5.5 Pro (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 97 % of answers carry at least one flag. Relative to answer length: 0,10 per 1000 output tokens.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5.5 Pro	78	71–84	-4	0,3	0,7	1,0	97 %	0,10	82 %
2	Claude Opus 4.7	77	72–81	+1	0,5	1,8	2,3	100 %	1,70	69 %
3	GPT-5	76	70–81	-3	0,6	0,9	1,5	100 %	0,62	65 %
4	Claude Opus 4.8	67	62–73	-4	0,6	1,0	1,6	100 %	1,08	63 %
5	Kimi K2.6	61	55–67	+2	0,8	1,9	2,7	100 %	0,31	52 %
6	Mistral Large 2	57	50–63	+1	0,7	1,3	2,1	93 %	3,41	53 %
7	Gemini 3.1 Pro	49	41–56	0	1,0	0,6	1,6	83 %	5,08	56 %
8	Grok 4.3	48	41–56	-3	0,7	0,8	1,5	87 %	5,09	57 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).