Ranking run · 2026-06-19

Ranking run 2026-06-19

Claude Opus 4.7 leads with 78/100 across all domains. Tax law: Claude Opus 4.7 (71) · Medicine: GPT-5.5 Pro (86) · Law: Claude Opus 4.7 (79) · Business law: GPT-5.5 Pro (82).

Status: development run — not in the feed, not on the homepage (roster test / interim run)
Run: run-20260619T195640-4ce8c8
Started: 2026-06-19 19:56 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 8
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.6
Pipeline cost: 106.01 USD
Judge agreement: 68 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	GPT-5.5 Pro n=72/100	28	164 / 266	80	76–84	—	83	0,12	—
2	Claude Opus 4.7	0	19 / 30	78	75–80	-2	59	1,52	—
3	Claude Opus 4.8	0	18 / 29	74	71–78	0	71	0,98	—
4	GPT-5	0	36 / 59	72	68–77	-4	68	0,65	—
5	Kimi K2.6	0	100 / 189	57	53–61	+1	48	0,27	—
6	Gemini 3.1 Pro	0	22 / 34	54	50–58	—	74	4,38	—
7	Mistral Large 2	0	10 / 67	52	47–56	+1	48	4,44	—
8	Grok 4.3	0	6 / 7	48	44–51	+4	75	4,89	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,6 times per answer and adds avg 1,6 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is GPT-5.5 Pro at avg 0,9 per answer. Relative to answer length: 1,80 per 1000 output tokens, lowest GPT-5.5 Pro at 0,14.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	71	65–78	+1	0,6	1,6	2,2	100 %	1,80	73 %
2	Claude Opus 4.8	67	58–74	-2	0,4	1,2	1,6	100 %	1,10	80 %
3	GPT-5.5 Pro	67	47–83	—	1,0	0,1	0,9	75 %	0,14	60 %
4	GPT-5	55	42–68	-7	1,1	1,2	2,3	93 %	0,85	65 %
5	Gemini 3.1 Pro	44	36–52	—	0,8	0,4	1,2	60 %	4,99	70 %
6	Kimi K2.6	33	22–43	-4	2,1	1,4	3,6	100 %	0,36	56 %
7	Grok 4.3	27	18–37	+4	1,5	0,7	2,1	93 %	8,87	59 %
8	Mistral Large 2	25	16–34	-3	2,3	1,3	3,5	100 %	7,58	50 %

Medicine · 20 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Medicine, GPT-5.5 Pro (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 0,4 claims the judge panel flags as false or unsupported; 85 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 0,4 per answer. Relative to answer length: 0,17 per 1000 output tokens.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5.5 Pro	86	81–90	—	0,5	0,4	0,7	85 %	0,17	73 %
2	Claude Opus 4.7	83	76–90	-2	0,5	0,9	1,3	95 %	1,33	82 %
3	Claude Opus 4.8	82	77–87	+3	0,3	0,4	0,6	85 %	0,59	81 %
4	GPT-5	80	73–87	-4	0,4	0,6	0,9	95 %	0,68	83 %
5	Mistral Large 2	71	65–78	+3	0,4	0,6	1,0	95 %	2,02	70 %
6	Kimi K2.6	64	55–71	-2	0,6	0,6	1,1	90 %	0,23	62 %
7	Gemini 3.1 Pro	60	54–67	—	0,2	0,3	0,4	60 %	1,85	71 %
8	Grok 4.3	56	49–63	+3	0,3	0,2	0,4	55 %	2,11	69 %

Law · 35 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 80 % of answers carry at least one flag. The lowest value is Grok 4.3 at avg 0,6 per answer. Relative to answer length: 1,30 per 1000 output tokens, lowest GPT-5.5 Pro at 0,09.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	79	73–84	-5	0,3	0,8	1,0	80 %	1,30	80 %
2	GPT-5.5 Pro	77	71–83	—	0,2	0,4	0,6	71 %	0,09	76 %
3	Claude Opus 4.8	76	70–82	-2	0,5	0,4	0,8	63 %	0,84	72 %
4	GPT-5	70	62–76	-4	0,5	0,5	1,0	86 %	0,60	69 %
5	Kimi K2.6	61	55–68	+4	0,7	0,7	1,4	89 %	0,17	67 %
6	Gemini 3.1 Pro	59	52–65	—	0,5	0,3	0,8	60 %	4,09	70 %
7	Mistral Large 2	49	40–56	+5	1,1	0,7	1,7	83 %	5,70	64 %
8	Grok 4.3	48	42–54	+6	0,5	0,2	0,6	54 %	4,36	67 %

Business law · 20 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Business law, GPT-5.5 Pro (top score) contradicts a stored key fact on average 0,1 times per answer and adds avg 0,6 claims the judge panel flags as false or unsupported; 90 % of answers carry at least one flag. Relative to answer length: 0,11 per 1000 output tokens.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5.5 Pro	82	73–89	—	0,1	0,6	0,7	90 %	0,11	76 %
2	GPT-5	79	74–85	-1	0,5	0,9	1,3	97 %	0,56	68 %
3	Claude Opus 4.7	76	71–80	-2	0,4	1,8	2,3	97 %	1,64	63 %
4	Claude Opus 4.8	71	65–77	+2	0,6	1,2	1,8	97 %	1,19	65 %
5	Kimi K2.6	59	52–65	+1	0,9	1,8	2,7	93 %	0,33	57 %
6	Mistral Large 2	56	50–62	-2	0,9	1,6	2,5	93 %	3,89	52 %
7	Grok 4.3	51	45–57	0	0,6	0,7	1,3	87 %	4,80	60 %
8	Gemini 3.1 Pro	49	41–56	—	1,1	0,8	1,7	93 %	5,50	56 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).