Ranking run · 2026-07-19

Ranking run 2026-07-19

Claude Opus 4.7 leads with 80/100 across all domains. Tax law: Claude Opus 4.7 (78) · Medicine: Claude Opus 4.7 (83) · Law: Claude Opus 4.7 (80) · Business law: Claude Opus 4.7 (80).

Status: official run (feed, archive, homepage)
Run: run-20260719T120108-c40839
Started: 2026-07-19 12:01 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 9
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.6
Pipeline cost: 121.36 USD
Judge agreement: 70 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 4.7	0	20 / 31	80	78–83	+2	61	1,36	—
2	Kimi K3 n=84/100	16 (12 T)	169 / 289	77	72–82	—	44	0,38	—
3	GPT-5.6 Sol	0	75 / 207	76	72–80	—	66	0,28	—
4	Claude Opus 4.8	0	19 / 27	75	72–78	0	72	0,90	—
4	GPT-5	0	24 / 38	75	71–78	0	69	0,64	—
5	Grok 4.5	0	31 / 58	63	59–66	—	81	2,52	—
6	Gemini 3.1 Pro	0	20 / 27	54	50–58	-3	72	4,59	—
7	Mistral Large 2	0	9 / 15	53	48–57	+3	52	4,13	—
8	GLM-4.6	0	74 / 163	47	43–51	+2	67	0,45	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–5 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 1,9 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,2 per answer. Relative to answer length: 1,72 per 1000 output tokens, lowest GPT-5.6 Sol at 0,27.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	78	70–85	+4	0,5	1,9	2,4	100 %	1,72	78 %
2	Claude Opus 4.8	69	63–75	+2	0,6	1,1	1,6	100 %	1,17	73 %
3	GPT-5.6 Sol	69	53–81	—	0,9	1,3	2,1	100 %	0,27	73 %
4	Kimi K3	61	42–77	—	1,1	2,8	3,9	100 %	0,35	64 %
5	GPT-5	59	46–73	+3	1,1	1,2	2,4	100 %	0,93	68 %
6	Grok 4.5	56	48–64	—	0,7	0,5	1,2	80 %	3,43	67 %
7	Gemini 3.1 Pro	41	32–50	-13	1,0	0,5	1,5	73 %	6,40	66 %
8	Mistral Large 2	30	19–42	+3	1,9	1,3	3,3	100 %	6,68	53 %
9	GLM-4.6	22	14–30	+3	1,8	0,9	2,6	100 %	0,71	49 %

Medicine · 20 Items

Ranks 1–6 statistically tied (95 % CI overlaps rank 1)

In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 0,8 claims the judge panel flags as false or unsupported; 95 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,3 per answer. Relative to answer length: 1,09 per 1000 output tokens, lowest GLM-4.6 at 0,12.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	83	77–89	+2	0,4	0,8	1,1	95 %	1,09	86 %
2	Kimi K3	83	76–90	—	0,6	0,8	1,4	90 %	0,41	83 %
3	GPT-5	82	73–88	-1	0,5	0,4	0,8	80 %	0,62	75 %
4	GPT-5.6 Sol	81	76–87	—	0,5	0,4	0,8	80 %	0,49	69 %
5	Claude Opus 4.8	80	73–86	-1	0,4	0,4	0,8	80 %	0,71	72 %
6	Grok 4.5	73	66–79	—	0,3	0,2	0,3	60 %	1,15	75 %
7	Mistral Large 2	69	63–75	+1	0,3	0,6	0,9	95 %	1,80	66 %
8	Gemini 3.1 Pro	62	56–68	+3	0,2	0,3	0,5	65 %	2,09	76 %
9	GLM-4.6	60	55–67	+3	0,1	0,3	0,3	55 %	0,12	73 %

Law · 35 Items

Ranks 1–5 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,3 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 91 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 0,5 per answer. Relative to answer length: 1,06 per 1000 output tokens, lowest GPT-5.6 Sol at 0,29.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	80	75–84	-2	0,3	0,7	0,9	91 %	1,06	80 %
2	Claude Opus 4.8	80	76–84	+1	0,3	0,4	0,6	71 %	0,69	77 %
3	Kimi K3	80	75–85	—	0,5	1,3	1,8	85 %	0,34	77 %
4	GPT-5	74	68–79	-4	0,3	0,7	1,0	86 %	0,56	74 %
5	GPT-5.6 Sol	72	65–79	—	0,6	0,6	1,1	86 %	0,29	70 %
6	Gemini 3.1 Pro	61	53–67	0	0,6	0,2	0,8	63 %	4,22	72 %
7	Grok 4.5	59	52–65	—	0,5	0,1	0,5	37 %	2,90	69 %
8	GLM-4.6	52	45–58	+7	0,5	0,5	0,9	63 %	0,36	67 %
9	Mistral Large 2	49	41–57	+5	1,1	0,6	1,6	71 %	5,44	64 %

Business law · 30 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Business law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,4 times per answer and adds avg 1,8 claims the judge panel flags as false or unsupported; 97 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,1 per answer. Relative to answer length: 1,53 per 1000 output tokens, lowest GPT-5.6 Sol at 0,25.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	80	76–84	+6	0,4	1,8	2,1	97 %	1,53	74 %
2	GPT-5.6 Sol	80	75–86	—	0,6	1,0	1,6	97 %	0,25	72 %
3	GPT-5	79	74–84	+3	0,6	0,7	1,3	97 %	0,56	72 %
4	Kimi K3	74	64–85	—	0,4	2,6	3,0	100 %	0,44	72 %
5	Claude Opus 4.8	68	62–74	-1	0,6	1,0	1,5	97 %	1,02	71 %
6	Grok 4.5	64	59–69	—	0,4	0,7	1,1	77 %	2,53	66 %
7	Mistral Large 2	57	50–64	+1	0,7	1,4	2,1	93 %	3,60	56 %
8	Gemini 3.1 Pro	48	40–56	-3	1,0	0,7	1,7	77 %	5,37	60 %
9	GLM-4.6	44	38–50	-5	1,0	0,9	1,9	87 %	0,55	54 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).