Ranking run · 2026-09-10

Ranking run 2026-09-10

Claude Opus 5 leads with 88/100 across all domains. Tax law: Claude Opus 5 (89) · Medicine: Claude Opus 5 (92) · Law: Claude Fable 5.1 (91) · Business law: Claude Opus 5 (80).

Status: official run (feed, archive, homepage)
Run: run-20260910T064154-bada0b
Started: 2026-09-10 06:41 UTC
Items: 126 (Tax law 31, Medicine 30, Law 35, Business law 30)
Models: 14
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-09-09-v5
Code version: 0.2.13
Pipeline cost: 283.50 USD
Judge agreement: 74 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 5	0	63 / 129	88	86–90	+2	25	0,56	+2,6
2	Claude Fable 5.1	0	49 / 91	83	80–86	+1	32	0,70	+2,5
3	Claude Opus 4.7	0	19 / 29	79	76–81	-1	46	1,84	+1,3
4	GPT-5.6 Sol	0	56 / 115	78	75–80	+2	69	0,57	-3,5
4	Kimi K3 n=103/126	23 (23 T)	146 / 272	78	74–81	0	36	0,47	0,0
5	GPT-5	0	67 / 137	76	72–80	-1	56	0,37	-2,5
5	GPT-6 Astra	0	40 / 76	76	73–79	+1	64	1,04	-4,2
6	Claude Opus 4.8	0	19 / 28	75	72–77	0	63	1,21	+1,9
6	GLM-5.3 n=82/126	44 (24 T)	126 / 409	75	70–79	-4	30	0,36	0,0
7	Grok 4.5	0	30 / 49	65	62–68	-1	79	2,59	0,0
8	Grok 4.6	0	65 / 124	63	59–66	0	71	3,40	0,0
9	DeepSeek V4 Pro	0	118 / 256	59	55–63	-1	66	0,21	0,0
10	Gemini 3.1 Pro	0	24 / 36	54	51–57	-2	78	3,74	0,0
11	Mistral Large 2	0	8 / 13	48	44–52	-2	42	5,00	+4,4

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 31 Items

Ranks 1–2 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 5 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 3,9 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 1,0 per answer. Relative to answer length: 0,59 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,26.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 5	89	85–92	+7	0,3	3,9	4,3	100 %	0,59	86 %
2	Claude Fable 5.1	81	74–87	+6	0,4	3,6	4,0	100 %	0,67	83 %
3	GPT-6 Astra	74	68–79	+5	0,3	1,6	1,9	100 %	0,98	72 %
4	Claude Opus 4.7	73	69–77	0	0,4	2,7	3,0	100 %	2,31	78 %
5	GPT-5.6 Sol	73	68–78	+5	0,3	1,5	1,8	100 %	0,52	78 %
6	Claude Opus 4.8	71	66–76	+2	0,3	1,7	2,0	100 %	1,38	78 %
7	GPT-5	65	57–73	+3	0,8	2,0	2,7	100 %	0,45	72 %
8	GLM-5.3	59	42–71	+13	1,3	3,8	5,1	100 %	0,54	73 %
9	Grok 4.5	58	53–63	+3	0,4	0,9	1,3	97 %	3,61	70 %
10	Kimi K3	58	43–71	-6	1,2	3,1	4,2	100 %	0,49	72 %
11	Grok 4.6	53	46–58	-7	0,6	1,3	1,9	97 %	4,72	69 %
12	Gemini 3.1 Pro	49	43–54	+4	0,4	0,6	1,0	74 %	4,22	67 %
13	DeepSeek V4 Pro	48	41–54	-2	0,8	1,5	2,3	94 %	0,26	65 %
14	Mistral Large 2	27	21–33	-2	1,4	1,9	3,3	100 %	7,22	53 %

Medicine · 30 Items

Ranks 1–7 statistically tied (95 % CI overlaps rank 1)

In Medicine, Claude Opus 5 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 1,9 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 0,4 per answer. Relative to answer length: 0,65 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,24.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 5	92	87–96	+2	0,2	1,9	2,1	100 %	0,65	90 %
2	Claude Fable 5.1	88	83–92	-2	0,2	1,3	1,5	100 %	0,71	84 %
3	GLM-5.3	87	82–91	+3	0,2	1,4	1,6	100 %	0,35	79 %
4	Kimi K3	87	81–91	0	0,2	1,3	1,6	100 %	0,50	84 %
5	Claude Opus 4.7	85	80–89	0	0,1	1,1	1,3	100 %	1,25	85 %
6	GPT-5	84	78–90	-1	0,3	1,0	1,3	100 %	0,41	83 %
7	GPT-5.6 Sol	83	79–88	0	0,2	0,6	0,8	90 %	0,78	78 %
8	GPT-6 Astra	82	78–85	-2	0,3	0,7	1,0	90 %	1,14	78 %
9	Claude Opus 4.8	81	75–86	0	0,2	0,7	0,9	97 %	0,87	79 %
10	Grok 4.5	70	65–76	0	0,1	0,3	0,5	77 %	1,59	70 %
11	DeepSeek V4 Pro	68	60–74	-2	0,2	0,3	0,5	80 %	0,24	72 %
12	Mistral Large 2	65	60–70	-3	0,3	1,2	1,5	100 %	2,85	69 %
13	Grok 4.6	64	58–70	+2	0,2	0,4	0,6	73 %	2,42	74 %
14	Gemini 3.1 Pro	60	54–65	0	0,1	0,3	0,4	57 %	1,90	73 %

Law · 35 Items

Ranks 1–2 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Fable 5.1 (top score) contradicts a stored key fact on average 0,0 times per answer and adds avg 1,7 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,6 per answer. Relative to answer length: 0,60 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,11.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Fable 5.1	91	88–93	+4	0,0	1,7	1,7	100 %	0,60	87 %
2	Claude Opus 5	91	88–94	+1	0,1	1,9	2,1	100 %	0,60	86 %
3	Kimi K3	81	76–86	+3	0,2	1,7	2,0	88 %	0,40	79 %
4	Claude Opus 4.8	80	76–84	-1	0,0	0,7	0,7	80 %	0,80	77 %
5	Claude Opus 4.7	79	73–85	-2	0,2	1,1	1,3	91 %	1,48	82 %
6	GPT-5	77	70–83	+2	0,2	1,0	1,2	86 %	0,28	77 %
7	GPT-5.6 Sol	77	72–82	+2	0,1	0,8	0,9	83 %	0,55	76 %
8	GPT-6 Astra	73	67–79	+2	0,3	1,0	1,3	91 %	1,23	69 %
9	GLM-5.3	70	60–79	-9	0,7	2,0	2,7	100 %	0,32	79 %
10	Grok 4.5	65	59–70	-3	0,2	0,4	0,6	51 %	2,84	70 %
11	Grok 4.6	64	56–71	+1	0,3	0,4	0,7	51 %	3,36	72 %
12	DeepSeek V4 Pro	61	54–68	+2	0,3	0,4	0,7	63 %	0,11	72 %
13	Gemini 3.1 Pro	59	52–64	-4	0,2	0,3	0,6	54 %	3,12	68 %
14	Mistral Large 2	43	35–51	-2	1,0	1,1	2,1	89 %	7,15	64 %

Business law · 30 Items

Ranks 1–9 statistically tied (95 % CI overlaps rank 1)

In Business law, Claude Opus 5 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 3,2 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,0 per answer. Relative to answer length: 0,47 per 1000 output tokens, lowest DeepSeek V4 Pro at 0,25.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 5	80	74–86	-1	0,5	3,2	3,7	100 %	0,47	79 %
2	GPT-5	80	72–86	-1	0,3	1,5	1,8	97 %	0,33	75 %
3	Claude Opus 4.7	78	73–82	-1	0,2	2,8	3,0	100 %	2,06	73 %
4	GPT-5.6 Sol	78	72–84	+2	0,3	1,1	1,5	100 %	0,56	73 %
5	GPT-6 Astra	75	69–82	-1	0,3	1,2	1,6	97 %	0,93	70 %
6	Kimi K3	74	65–82	+1	0,5	3,1	3,6	100 %	0,50	73 %
7	Claude Fable 5.1	73	65–81	0	0,6	3,2	3,7	100 %	0,81	71 %
8	Grok 4.6	69	62–75	+5	0,4	1,0	1,4	97 %	2,77	66 %
9	GLM-5.3	68	55–78	-11	0,6	3,6	4,2	100 %	0,35	73 %
10	Claude Opus 4.8	67	62–74	+1	0,5	1,9	2,4	100 %	1,56	69 %
11	Grok 4.5	67	61–71	-1	0,2	0,8	1,0	90 %	2,28	60 %
12	DeepSeek V4 Pro	59	52–67	+1	0,5	1,4	1,9	97 %	0,25	63 %
13	Mistral Large 2	59	52–66	+4	0,4	2,1	2,4	100 %	3,93	56 %
14	Gemini 3.1 Pro	49	42–56	-2	0,7	0,9	1,5	97 %	5,07	58 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).