Ranking run · 2026-09-01

Ranking run 2026-09-01

Claude Opus 4.7 leads with 78/100 across all domains. Tax law: Kimi K3 (77) · Medicine: GPT-5.6 Sol (85) · Law: Claude Opus 4.7 (82) · Business law: GPT-5 (79).

Status: development run — not in the feed, not on the homepage (roster test / interim run)
Run: run-20260901T060205-7d51d0
Started: 2026-09-01 06:02 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 9
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.6
Pipeline cost: 119.85 USD
Judge agreement: 69 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 4.7	0	19 / 29	78	75–81	-1	59	1,47	—
2	GPT-5.6 Sol	0	75 / 183	77	74–80	0	66	0,28	—
3	Kimi K3 n=79/100	21	144 / 527	76	71–81	—	46	0,42	—
4	Claude Opus 4.8	0	19 / 28	75	72–78	+1	71	0,96	—
5	GPT-5	0	21 / 35	72	68–77	-5	66	0,69	—
6	Grok 4.5	0	27 / 50	63	59–66	+5	83	2,11	—
7	Gemini 3.1 Pro	0	23 / 33	54	50–58	0	72	4,79	—
8	Mistral Large 2	0	8 / 14	51	46–55	+1	52	4,12	—
9	GLM-4.6	0	70 / 145	45	40–49	0	64	0,52	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 4 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Tax law, Kimi K3 (top score) contradicts a stored key fact on average 1,3 times per answer and adds avg 1,6 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,0 per answer. Relative to answer length: 0,34 per 1000 output tokens, lowest GPT-5.6 Sol at 0,34.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Kimi K3	77	67–84	—	1,3	1,6	2,8	100 %	0,34	66 %
2	Claude Opus 4.7	71	64–78	-1	0,7	1,8	2,4	100 %	1,87	79 %
3	GPT-5.6 Sol	68	54–77	-2	0,9	1,6	2,5	93 %	0,34	71 %
4	Claude Opus 4.8	65	59–71	0	0,7	1,1	1,8	100 %	1,30	66 %
5	Grok 4.5	58	51–64	+9	0,8	0,3	1,0	73 %	2,67	63 %
6	GPT-5	54	39–70	-11	1,3	1,2	2,5	100 %	0,99	63 %
7	Gemini 3.1 Pro	40	30–50	-4	1,1	0,7	1,8	73 %	7,78	72 %
8	Mistral Large 2	30	21–41	+6	1,7	1,2	2,8	100 %	5,74	46 %
9	GLM-4.6	24	16–33	+1	2,0	0,8	2,8	93 %	0,85	60 %

Medicine · 20 Items

Ranks 1–5 statistically tied (95 % CI overlaps rank 1)

In Medicine, GPT-5.6 Sol (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 0,4 claims the judge panel flags as false or unsupported; 80 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,3 per answer. Relative to answer length: 0,42 per 1000 output tokens, lowest GLM-4.6 at 0,15.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5.6 Sol	85	81–90	0	0,4	0,4	0,7	80 %	0,42	75 %
2	Claude Opus 4.7	83	75–89	-1	0,4	0,6	0,9	100 %	0,90	78 %
3	Kimi K3	81	74–88	—	0,8	0,7	1,4	90 %	0,49	73 %
4	Claude Opus 4.8	80	75–85	0	0,3	0,5	0,7	95 %	0,68	74 %
5	GPT-5	78	68–85	-6	0,5	0,6	1,1	85 %	0,79	81 %
6	Mistral Large 2	69	63–76	+3	0,4	0,7	1,1	95 %	2,19	68 %
7	Grok 4.5	66	59–73	+2	0,3	0,1	0,3	40 %	1,18	69 %
8	Gemini 3.1 Pro	62	55–69	-1	0,3	0,2	0,4	55 %	1,71	73 %
9	GLM-4.6	56	49–63	-4	0,1	0,3	0,4	65 %	0,15	67 %

Law · 35 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 80 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,4 per answer. Relative to answer length: 1,17 per 1000 output tokens, lowest GPT-5.6 Sol at 0,32.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	82	77–86	+1	0,2	0,8	1,0	80 %	1,17	84 %
2	Claude Opus 4.8	80	75–85	-1	0,2	0,5	0,7	77 %	0,78	84 %
3	GPT-5.6 Sol	75	67–81	0	0,5	0,7	1,1	86 %	0,32	75 %
4	Kimi K3	75	66–82	—	0,9	1,1	2,0	94 %	0,39	79 %
5	GPT-5	71	66–77	-2	0,4	0,7	1,1	86 %	0,67	70 %
6	Grok 4.5	65	59–70	+7	0,3	0,1	0,4	40 %	2,00	73 %
7	Gemini 3.1 Pro	61	53–66	+1	0,4	0,3	0,7	60 %	3,86	68 %
8	GLM-4.6	48	41–54	+3	0,6	0,4	1,1	66 %	0,46	65 %
9	Mistral Large 2	45	37–53	0	1,2	0,7	1,8	80 %	6,08	64 %

Business law · 30 Items

Ranks 1–5 statistically tied (95 % CI overlaps rank 1)

In Business law, GPT-5 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,9 claims the judge panel flags as false or unsupported; 93 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,1 per answer. Relative to answer length: 0,50 per 1000 output tokens, lowest GPT-5.6 Sol at 0,21.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5	79	71–85	-5	0,3	0,9	1,2	93 %	0,50	72 %
2	GPT-5.6 Sol	79	74–84	+1	0,6	0,8	1,4	97 %	0,21	71 %
3	Claude Opus 4.7	75	70–80	-3	0,5	2,0	2,5	100 %	1,77	70 %
4	Kimi K3	75	64–85	—	0,6	2,4	3,0	100 %	0,45	72 %
5	Claude Opus 4.8	69	63–75	+3	0,5	1,1	1,6	97 %	1,08	66 %
6	Grok 4.5	61	54–69	+3	0,5	0,6	1,1	73 %	2,26	63 %
7	Mistral Large 2	58	52–63	+1	0,6	1,5	2,0	100 %	3,40	50 %
8	Gemini 3.1 Pro	49	41–56	+3	1,1	0,7	1,8	90 %	5,87	59 %
9	GLM-4.6	43	35–50	-1	1,0	0,9	1,9	93 %	0,60	58 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).