Ranking run · 2026-09-04

Ranking run 2026-09-04

Claude Opus 5 leads with 86/100 across all domains. Tax law: Claude Opus 5 (85) · Medicine: Claude Opus 5 (91) · Law: Claude Opus 5 (91) · Business law: Claude Opus 5 (79).

Status: development run — not in the feed, not on the homepage (roster test / interim run)
Run: run-20260904T151155-c623e0
Started: 2026-09-04 15:11 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 14
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.7
Pipeline cost: 233.44 USD
Judge agreement: 70 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 5	0	62 / 130	86	83–89	—	48	0,39	—
2	Claude Fable 5.1 n=98/100	2 (1 T)	49 / 89	81	77–85	—	40	0,61	—
3	Claude Opus 4.7 n=99/100	1 (1 T)	20 / 29	79	75–82	+1	60	1,41	—
4	Kimi K3 n=81/100	19 (18 T)	146 / 286	78	74–83	+2	50	0,35	—
5	GLM-5.3 n=68/100	32	139 / 409	77	71–82	—	47	0,26	—
6	Claude Opus 4.8	0	19 / 30	76	73–79	+1	68	1,08	—
6	GPT-5.6 Sol	0	71 / 181	76	72–79	-1	67	0,27	—
7	GPT-6 Astra	0	109 / 214	75	71–78	—	63	0,37	—
8	GPT-5	0	32 / 50	74	69–78	+2	66	0,68	—
9	Grok 4.6	0	68 / 115	65	62–69	—	78	2,63	—
10	Grok 4.5	0	31 / 52	64	61–68	+1	83	2,08	—
11	DeepSeek V4 Pro n=98/100	2	99 / 207	59	55–63	—	71	0,18	—
12	Gemini 3.1 Pro n=96/100	4 (4 T)	27 / 101	55	51–58	+1	74	4,36	—
13	Mistral Large 2	0	8 / 15	50	45–55	-1	47	4,64	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–6 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 5 (top score) contradicts a stored key fact on average 0,7 times per answer and adds avg 2,3 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,2 per answer. Relative to answer length: 0,38 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,31.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 5	85	76–94	—	0,7	2,3	3,0	100 %	0,38	80 %
2	Claude Fable 5.1	76	64–86	—	0,9	2,8	3,7	100 %	0,56	75 %
3	GLM-5.3	76	67–81	—	0,7	2,5	3,2	100 %	0,39	78 %
4	Claude Opus 4.7	73	65–82	+2	0,5	1,8	2,2	93 %	1,66	65 %
5	Claude Opus 4.8	70	61–77	+5	0,7	1,0	1,6	100 %	1,17	77 %
6	GPT-6 Astra	68	58–77	—	1,0	1,3	2,3	100 %	0,40	74 %
7	GPT-5.6 Sol	61	53–71	-7	1,3	1,5	2,7	100 %	0,33	53 %
8	GPT-5	59	43–72	+5	1,5	0,9	2,5	100 %	0,88	60 %
9	Kimi K3	56	34–76	-21	1,4	1,6	3,0	100 %	0,35	71 %
10	Grok 4.6	54	47–62	—	1,1	0,7	1,8	93 %	4,51	59 %
11	Grok 4.5	50	42–60	-8	0,8	0,5	1,2	87 %	3,54	59 %
12	Gemini 3.1 Pro	46	37–55	+6	1,1	0,7	1,7	86 %	6,86	60 %
13	DeepSeek V4 Pro	41	29–54	—	1,8	1,1	2,8	92 %	0,31	56 %
14	Mistral Large 2	28	19–39	-2	2,2	1,4	3,5	93 %	7,44	55 %

Medicine · 20 Items

Ranks 1–9 statistically tied (95 % CI overlaps rank 1)

In Medicine, Claude Opus 5 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 1,1 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 0,4 per answer. Relative to answer length: 0,45 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,17.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 5	91	85–96	—	0,4	1,1	1,5	100 %	0,45	89 %
2	Claude Fable 5.1	85	77–92	—	0,5	0,8	1,3	100 %	0,61	84 %
3	GLM-5.3	84	77–90	—	0,6	0,9	1,3	100 %	0,27	82 %
4	GPT-5	84	78–90	+6	0,3	0,6	0,8	95 %	0,59	82 %
5	GPT-6 Astra	84	81–88	—	0,5	0,5	0,9	95 %	0,49	78 %
6	Claude Opus 4.7	83	76–89	0	0,4	0,7	1,0	95 %	0,92	72 %
7	Claude Opus 4.8	83	77–88	+3	0,3	0,7	0,9	100 %	0,80	78 %
8	Kimi K3	83	77–90	+2	0,7	0,8	1,3	90 %	0,37	74 %
9	GPT-5.6 Sol	82	77–87	-3	0,6	0,3	0,7	85 %	0,44	69 %
10	DeepSeek V4 Pro	70	63–78	—	0,2	0,3	0,4	65 %	0,17	66 %
11	Grok 4.6	68	61–75	—	0,3	0,2	0,4	65 %	1,81	68 %
12	Grok 4.5	67	59–75	+1	0,3	0,2	0,4	60 %	1,63	71 %
13	Mistral Large 2	65	60–70	-4	0,6	0,8	1,4	95 %	2,83	66 %
14	Gemini 3.1 Pro	61	54–68	-1	0,3	0,2	0,4	50 %	1,56	73 %

Law · 35 Items

Ranks 1–2 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 5 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 1,3 claims the judge panel flags as false or unsupported; 94 % of answers carry at least one flag. The lowest value is Grok 4.6 at avg 0,4 per answer. Relative to answer length: 0,47 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,12.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 5	91	87–94	—	0,3	1,3	1,7	94 %	0,47	84 %
2	Claude Fable 5.1	88	83–92	—	0,2	1,3	1,5	94 %	0,50	86 %
3	Claude Opus 4.7	80	73–87	-2	0,2	0,7	0,9	79 %	1,04	82 %
4	Claude Opus 4.8	80	76–84	0	0,3	0,6	0,9	80 %	0,96	78 %
5	Kimi K3	80	72–86	+5	0,5	1,1	1,6	88 %	0,29	79 %
6	GPT-5.6 Sol	76	70–82	+1	0,3	0,7	0,9	83 %	0,24	72 %
7	GLM-5.3	74	64–82	—	0,8	1,2	2,1	97 %	0,24	76 %
8	GPT-5	72	64–78	+1	0,7	0,6	1,2	80 %	0,73	72 %
9	GPT-6 Astra	71	64–77	—	0,9	0,6	1,3	94 %	0,41	70 %
10	Grok 4.6	68	62–74	—	0,2	0,2	0,4	43 %	1,92	72 %
11	Grok 4.5	67	61–72	+2	0,3	0,2	0,4	51 %	2,18	71 %
12	DeepSeek V4 Pro	61	54–68	—	0,5	0,3	0,8	66 %	0,12	68 %
13	Gemini 3.1 Pro	59	52–65	-2	0,5	0,3	0,7	68 %	3,81	68 %
14	Mistral Large 2	46	38–54	+1	1,1	0,8	1,9	83 %	6,35	67 %

Business law · 30 Items

Ranks 1–9 statistically tied (95 % CI overlaps rank 1)

In Business law, Claude Opus 5 (top score) contradicts a stored key fact on average 0,7 times per answer and adds avg 1,8 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,8 per answer. Relative to answer length: 0,34 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,18.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 5	79	72–86	—	0,7	1,8	2,5	100 %	0,34	65 %
2	Claude Opus 4.7	78	73–83	+3	0,5	2,0	2,5	100 %	1,80	72 %
3	GPT-5.6 Sol	78	73–83	-1	0,6	0,8	1,4	100 %	0,22	70 %
4	Kimi K3	78	68–86	+3	0,6	2,3	2,9	100 %	0,42	70 %
5	GPT-5	77	72–83	-2	0,5	0,8	1,3	93 %	0,55	63 %
6	GPT-6 Astra	76	71–82	—	0,7	0,8	1,5	100 %	0,29	67 %
7	Claude Fable 5.1	74	66–81	—	0,9	2,6	3,5	100 %	0,72	72 %
8	GLM-5.3	72	59–84	—	0,4	2,7	3,1	100 %	0,26	72 %
9	Claude Opus 4.8	69	64–75	0	0,6	1,4	1,9	97 %	1,25	69 %
10	Grok 4.5	66	61–72	+5	0,3	0,5	0,8	73 %	1,66	61 %
11	Grok 4.6	65	59–71	—	0,6	0,7	1,2	93 %	2,47	65 %
12	DeepSeek V4 Pro	57	49–62	—	0,7	0,7	1,3	90 %	0,18	64 %
13	Mistral Large 2	56	48–63	-2	0,7	1,5	2,2	97 %	3,59	56 %
14	Gemini 3.1 Pro	49	42–55	0	1,1	0,6	1,6	100 %	5,21	57 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).