Ranking run · 2026-08-01

Ranking run 2026-08-01

Claude Opus 4.7 leads with 81/100 across all domains. Tax law: Claude Opus 4.7 (75) · Medicine: Claude Opus 4.7 (87) · Law: Claude Opus 4.7 (84) · Business law: GPT-5.6 Sol (79).

Status: official run (feed, archive, homepage)
Run: run-20260801T055440-452791
Started: 2026-08-01 05:54 UTC
Items: 100 (Tax law 15, Medicine 20, Law 35, Business law 30)
Models: 9
Judges: GPT-5, Claude Opus 4.8, Claude Opus 4.7, Mistral Large 2
Question bank: 2026-05-17-v1
Code version: 0.2.6
Pipeline cost: 119.06 USD
Judge agreement: 70 %

Overall across domains

#	Model	top100	95 % CI	Δ prev.	failures	sec median/p90	hallu band	per 1000 tok.	Δ own family
1	Claude Opus 4.7	0	20 / 31	81	78–84	+1	61	1,36	—
2	Kimi K3 n=71/100	29 (27 T)	159 / 413	80	76–84	+3	52	0,38	—
3	GPT-5.6 Sol	0	89 / 208	77	73–80	+1	67	0,26	—
4	Claude Opus 4.8	0	20 / 31	75	72–78	0	69	1,04	—
5	GPT-5	0	25 / 43	73	69–77	-2	66	0,67	—
6	Grok 4.5	0	29 / 59	59	55–63	-4	78	3,03	—
7	Gemini 3.1 Pro	0	20 / 37	56	52–60	+2	76	3,98	—
8	Mistral Large 2	0	9 / 17	51	46–56	-2	46	4,60	—
9	GLM-4.6 n=96/100	4 (4 T)	60 / 137	46	41–49	-1	69	0,44	—

Δ prev. = difference to the previous published run; “—” = model was not in it. Hallu band: 100 = none, 0 = ≥ 4 false claims per answer. Per 1000 tok. = flagged claims per 1000 output tokens (length-normalised). Δ own family = score with all judges minus score with foreign judges only, in points; positive = own family scores more generously; “—” = not recorded before code_version 0.2.8 or no foreign judge (methodology §4, §6.5).

Tax law · 15 Items

Ranks 1–5 statistically tied (95 % CI overlaps rank 1)

In Tax law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 1,6 claims the judge panel flags as false or unsupported; 100 % of answers carry at least one flag. The lowest value is Gemini 3.1 Pro at avg 1,3 per answer. Relative to answer length: 1,60 per 1000 output tokens, lowest GPT-5.6 Sol at 0,27.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	75	70–81	-3	0,5	1,6	2,2	100 %	1,60	72 %
2	GPT-5.6 Sol	71	62–79	+2	0,9	1,3	2,2	100 %	0,27	73 %
3	Claude Opus 4.8	66	58–74	-3	0,8	0,8	1,5	100 %	1,13	70 %
4	GPT-5	63	51–74	+4	1,0	1,3	2,3	93 %	0,81	69 %
5	Kimi K3	55	47–70	-6	2,0	2,3	4,3	100 %	0,49	71 %
6	Grok 4.5	52	42–60	-4	0,7	0,8	1,5	87 %	4,46	60 %
7	Gemini 3.1 Pro	47	37–57	+6	0,7	0,5	1,3	67 %	5,32	69 %
8	Mistral Large 2	29	20–39	-1	2,1	1,2	3,3	93 %	6,67	49 %
9	GLM-4.6	22	14–31	0	1,7	0,9	2,5	93 %	0,75	60 %

Medicine · 20 Items

Ranks 1–5 statistically tied (95 % CI overlaps rank 1)

In Medicine, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,5 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 90 % of answers carry at least one flag. The lowest value is GLM-4.6 at avg 0,2 per answer. Relative to answer length: 1,07 per 1000 output tokens, lowest GLM-4.6 at 0,08.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	87	80–93	+4	0,5	0,7	1,2	90 %	1,07	81 %
2	GPT-5.6 Sol	84	80–89	+3	0,4	0,4	0,8	80 %	0,44	80 %
3	Kimi K3	82	74–90	-1	0,6	0,8	1,4	100 %	0,41	83 %
4	GPT-5	80	72–87	-2	0,4	0,7	1,1	95 %	0,85	77 %
5	Claude Opus 4.8	79	72–85	-1	0,5	0,6	1,0	100 %	0,93	77 %
6	Mistral Large 2	71	65–76	+2	0,3	0,8	1,1	95 %	2,25	67 %
7	Gemini 3.1 Pro	66	61–71	+4	0,2	0,3	0,4	60 %	1,62	73 %
8	Grok 4.5	63	55–71	-10	0,3	0,2	0,4	50 %	1,36	64 %
9	GLM-4.6	57	50–64	-3	0,1	0,1	0,2	37 %	0,08	81 %

Law · 35 Items

Ranks 1–4 statistically tied (95 % CI overlaps rank 1)

In Law, Claude Opus 4.7 (top score) contradicts a stored key fact on average 0,2 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 80 % of answers carry at least one flag. The lowest value is Claude Opus 4.8 at avg 0,7 per answer. Relative to answer length: 1,03 per 1000 output tokens, lowest GPT-5.6 Sol at 0,29.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	Claude Opus 4.7	84	80–88	+4	0,2	0,7	0,9	80 %	1,03	85 %
2	Kimi K3	82	77–87	+2	0,6	1,0	1,5	94 %	0,31	79 %
3	Claude Opus 4.8	81	76–85	+1	0,1	0,5	0,7	80 %	0,72	80 %
4	GPT-5.6 Sol	73	65–80	+1	0,5	0,7	1,2	86 %	0,29	75 %
5	GPT-5	72	65–78	-2	0,5	0,6	1,0	80 %	0,59	72 %
6	Gemini 3.1 Pro	59	52–64	-2	0,5	0,2	0,7	57 %	3,76	60 %
7	Grok 4.5	56	50–63	-3	0,7	0,2	0,8	66 %	4,24	68 %
8	GLM-4.6	49	42–55	-3	0,5	0,5	0,9	67 %	0,39	66 %
9	Mistral Large 2	43	35–51	-6	1,3	0,8	2,2	74 %	7,04	67 %

Business law · 30 Items

Ranks 1–5 statistically tied (95 % CI overlaps rank 1)

In Business law, GPT-5.6 Sol (top score) contradicts a stored key fact on average 0,7 times per answer and adds avg 0,7 claims the judge panel flags as false or unsupported; 97 % of answers carry at least one flag. The lowest value is Grok 4.5 at avg 1,0 per answer. Relative to answer length: 0,21 per 1000 output tokens.

#	Model	top100	95 % CI	Δ prev.	contradicts key fact	extra false/unsupp.	avg total	answers affected	per 1000 tok.	IRR
1	GPT-5.6 Sol	79	73–84	-1	0,7	0,7	1,4	97 %	0,21	66 %
2	Kimi K3	79	68–89	+5	0,5	2,4	2,8	100 %	0,43	70 %
3	Claude Opus 4.7	76	72–81	-4	0,3	2,0	2,3	100 %	1,63	68 %
4	GPT-5	75	67–82	-4	0,6	0,9	1,4	100 %	0,59	69 %
5	Claude Opus 4.8	71	65–77	+3	0,5	1,4	1,9	93 %	1,27	69 %
6	Grok 4.5	63	58–68	-1	0,5	0,5	1,0	87 %	2,50	60 %
7	Mistral Large 2	59	52–66	+2	0,9	1,4	2,3	93 %	3,62	58 %
8	Gemini 3.1 Pro	52	44–59	+4	0,9	0,6	1,5	87 %	4,82	61 %
9	GLM-4.6	47	41–52	+3	0,8	0,8	1,6	93 %	0,47	59 %

Raw data of this run

raw.jsonl and progress.jsonl are the public streams with response texts stripped (contamination protection, methodology §7).