Ranking run · 2026-07-01

Ranking run 2026-07-01

Claude Opus 4.7 leads with 80/100 across all domains. Tax law: Claude Opus 4.7 (73) · Medicine: GPT-5.5 Pro (88) · Law: Claude Opus 4.7 (82) · Business law: GPT-5 (82).

Status: development run — not in the feed, not on the homepage (roster test / interim run)
Run: run-20260701T043526-2a195e
Started: 2026-07-01 04:35 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 9
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.6
Pipeline cost: 125.26 USD
Judge agreement: 67 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 4.7	0	19 / 30	80	76–83	0	58	1,51	—
1	GPT-5.5 Pro n=83/100	17 (17 T)	223 / 771	80	76–83	+2	77	0,11	—
2	GPT-5	0	37 / 54	76	72–80	0	65	0,71	—
3	Claude Opus 4.8	0	18 / 27	75	72–78	+1	73	0,87	—
4	Kimi K2.6	0	189 / 519	59	55–64	0	47	0,30	—
5	Gemini 3.1 Pro	0	20 / 29	56	52–60	+2	75	4,10	—
6	Mistral Large 2	0	11 / 69	51	47–55	0	48	4,51	—
7	GLM-4.6	0	72 / 116	47	43–51	—	68	0,45	—
8	Grok 4.3	0	10 / 13	45	40–49	-3	73	5,38	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 1,9 claims the judge panel flags as false or unsupported; 93 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 1,2 per answer. Relative to answer length: 1,85 per 1000 output tokens, lowest GPT-5.5 Pro at 0,19.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	73	61–80	-5	0,4	1,9	2,3	93 %	1,85	81 %
2	Claude Opus 4.8	66	60–71	+1	0,5	0,8	1,4	87 %	0,99	64 %
3	GPT-5	60	46–71	-7	1,5	0,8	2,3	100 %	0,89	62 %
4	GPT-5.5 Pro	57	43–72	-11	1,7	0,8	2,3	100 %	0,19	48 %
5	Gemini 3.1 Pro	48	41–55	+4	0,9	0,3	1,2	73 %	4,97	69 %
6	Kimi K2.6	31	20–42	+1	1,9	1,8	3,7	100 %	0,39	50 %
7	Mistral Large 2	27	19–35	0	1,8	1,3	3,1	100 %	6,54	54 %
8	Grok 4.3	25	17–33	-3	1,7	0,4	2,0	87 %	9,07	56 %
9	GLM-4.6	21	13–30	—	1,8	0,8	2,5	100 %	0,80	51 %

Medicine · 20 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Medicine, GPT-5.5 Pro (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,3 claims the judge panel flags as false or unsupported; 70 % of answers carry at least one flag. The lowest value is GLM-4.6 at avg 0,3 per answer. Relative to answer length: 0,14 per 1000 output tokens, lowest GLM-4.6 at 0,12.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5.5 Pro	88	84–92	+1	0,3	0,3	0,6	70 %	0,14	81 %
2	Claude Opus 4.7	84	80–89	+1	0,3	0,8	1,1	100 %	1,04	81 %
3	GPT-5	82	72–90	+4	0,6	0,6	1,1	85 %	0,80	80 %
4	Claude Opus 4.8	80	75–86	-1	0,3	0,6	0,8	90 %	0,79	79 %
5	Kimi K2.6	74	68–82	+4	0,5	0,6	0,9	90 %	0,18	75 %
6	Gemini 3.1 Pro	65	59–71	+2	0,3	0,2	0,4	60 %	1,60	72 %
7	Mistral Large 2	65	59–72	-4	0,5	0,6	1,1	90 %	2,25	65 %
8	GLM-4.6	59	52–65	—	0,2	0,2	0,3	45 %	0,12	70 %
9	Grok 4.3	57	51–63	-3	0,1	0,3	0,4	55 %	1,92	72 %

Law · 35 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 80 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 0,6 per answer. Relative to answer length: 1,19 per 1000 output tokens, lowest GPT-5.5 Pro at 0,09.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	82	77–87	0	0,2	0,8	1,0	80 %	1,19	80 %
2	Claude Opus 4.8	82	76–87	+2	0,2	0,5	0,6	69 %	0,64	77 %
3	GPT-5.5 Pro	76	70–82	-1	0,2	0,6	0,8	81 %	0,09	73 %
4	GPT-5	74	67–80	-3	0,5	0,6	1,1	80 %	0,64	77 %
5	Kimi K2.6	65	60–70	+2	0,6	0,8	1,4	83 %	0,21	70 %
6	Gemini 3.1 Pro	61	54–67	+3	0,3	0,3	0,6	46 %	3,26	64 %
7	GLM-4.6	51	44–58	—	0,4	0,4	0,9	63 %	0,32	70 %
8	Mistral Large 2	49	41–57	+4	1,1	0,9	2,0	86 %	6,34	66 %
9	Grok 4.3	46	40–52	-3	0,8	0,2	0,9	51 %	7,56	63 %

Business law · 30 Items

Ranks 1–3 statistically tied (95 % CI overlaps rank 1)

In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 1,1 claims the judge panel flags as false or unsupported; 97 % of answers carry at least one flag. The lowest value is GPT-5.5 Pro at avg 1,0 per answer. Relative to answer length: 0,64 per 1000 output tokens, lowest GPT-5.5 Pro at 0,10.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5	82	77–87	+6	0,4	1,1	1,5	97 %	0,64	75 %
2	GPT-5.5 Pro	82	78–87	+4	0,4	0,6	1,0	92 %	0,10	73 %
3	Claude Opus 4.7	77	72–83	0	0,6	1,9	2,5	100 %	1,80	72 %
4	Claude Opus 4.8	69	64–74	+2	0,6	1,1	1,6	97 %	1,02	65 %
5	Kimi K2.6	57	50–64	-4	0,7	2,1	2,9	100 %	0,40	59 %
6	Mistral Large 2	56	49–63	-1	0,9	1,5	2,3	100 %	3,78	50 %
7	Gemini 3.1 Pro	49	40–57	0	1,0	0,7	1,7	90 %	5,58	56 %
8	GLM-4.6	48	41–54	—	0,9	1,0	1,9	97 %	0,56	59 %
9	Grok 4.3	46	38–54	-2	0,7	0,6	1,3	80 %	4,47	53 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).