Methodology

How the leaderboard comes about.

This page describes the evaluation procedure in full. The standalone pipeline produces a reproducible JSONL dataset for each run; this is publicly accessible in the audit trail and replaces any claim that is not backed by data.

How to read the leaderboard — the most important rule: Every score carries a 95% bootstrap confidence interval. If two models overlap in their CI bands (median ±3 points at 126 items in the current run), they are shown as statistically tied — same rank position, same bar height, shared caption. An ordering is only asserted once the CI bands no longer overlap. Details below under "Bootstrap confidence intervals".

Bootstrap confidence intervals & rank ties

For each (model × domain), 1000 bootstrap resamples are drawn on the cell scores, and the 95% percentile interval is published. No point value without an uncertainty figure — a score difference that lies within the CI bands is not statistically distinguishable and is therefore not asserted as a ranking difference either.

Concretely: the current run covers 126 items (Tax law 31 · Medicine 30 · Law 35 · Business law 30). The half-width of the 95 % bootstrap CI has a median of ±3 points across models overall, per domain: Tax law ±6 (N=31), Medicine ±5 (N=30), Law ±6 (N=35), Business law ±7 (N=30). Gaps below half the CI width of a domain are not reliable there; the rankings mark such groups as “statistically tied”. Two models with overlapping CI bands share a rank on the ranking page only when their top100 score is exactly identical; otherwise the order stands, but the gap is not reliable within the CI. As the item bank grows the bands narrow — the item count per domain is on every run page under /en-runs.

Recommendation under incomplete coverage: When a model misses items because of technical failures (timeouts, empty responses) the ranking stays exactly as measured — only the recommendation badge follows two tests. First: a model that answered at least 95 % of the items is eligible without penalty; a handful of transport errors is not a quality signal. Second, for models below that: score every missing item pessimistically as 0 — if the model still leads the best fully-covered model, the lead is not an artefact of the gap and the recommendation stays with the leader. Only when both tests fail does the badge move to the best model with sufficient coverage — with the reason and both numbers right under the podium. Until September 2026 the rule was a rigid "full coverage only", which in the 10 September run would have lifted a model ten points weaker onto the recommendation because of a single missing item.

Consequence for the reader: When the ranking chart shows two models as tied, then either of the two is an equally well-substantiated choice. Anyone who needs a hard tie-break rule will find the hallucination rate and the inter-rater agreement per cell in the audit trail — both tie-breakers that do not rest on a spurious precision of the mean.

The following sections document the procedure in detail. Each item is collapsed — click to expand, or follow a direct anchor link (e.g. /methodik#triple-judge). Anchor links open the respective section automatically.

How is the question bank structured per domain?

Each of the four domains has a versioned question bank in the repository. The bank separates two kinds:

Public items — derived directly from accessible sources (BFH rulings, BGH decisions, medical guidelines, standard legal literature). These measure how well a model masters established content.
Synthetic items — created by a proposer model, critiqued by a reviewer model, human-approved by the owner. Realistic mandate cases with interwoven factors — not textbook exercises. These measure how well a model handles unfamiliar constellations, and eliminate the training effect as well as the tool-search advantage.

Question-bank versioning via Git: a second run against the same question_bank_version produces byte-identical answers. Methodology changes trigger a new version; old data points remain visible under their original version.

Synthetic items stay private — deliberately. The item prompts (case constellations, target facts, scoring rubrics) reside exclusively in a private repository and are never published. If the question catalog became public, model vendors could include the items in their next training dataset — the ranking would tip from genuine generalization ability to memorization. In the published raw.jsonl, the model answer texts and the verbatim extracted hallucinations are also removed, because the original items would be reconstructable from them. The aggregated cross-domain validation results, by contrast, are fully open in the public validation gist (10 files: Methodology, Legal Study, Medical Study, Limitations, Critique Response, …) — no single item text contained within. Raw answers are available exclusively on direct NDA request.

Which models are evaluated — and what is the tool configuration?

All enabled flagship models from the four providers are evaluated — typically Anthropic, OpenAI, Google, Mistral. Plug-in pattern via models.json: new providers are activated by a config entry, no code intervention needed.

Per cell, the model is tested twice: once solo (no tool access) and once with the tool registry enabled (web_search, doc_retrieval, pubmed_search, arxiv_search, url_fetch). Both series are published separately, because the model ordering can shift noticeably between the two modes.

More importantly: the best model changes from question to question — even within the same domain. Why a single ranking percentage is therefore misleading (and what the "oracle" / best-per-question means): Ranking → No model leads everywhere.

How is scoring done — closed items, open items, hallucination detection?

Items have two answer types:

Closed items expect a concrete answer (number, multiple choice, exact value). Scored via regex or range match: 1 or 0 per item.
Open items expect a fully worded justification. Scored by a dedicated open-rubric judge that, for every recorded target fact, decides: entail (contained in the model output), missing (absent) or contradict (contradicts).

In addition, the judge records in every answer extra false claims — freely invented assertions beyond the target rubric. Invented case numbers, wrong paragraph numbers, fabricated studies, wrong figures are extracted verbatim, archived internally, and published in aggregated form per run as the hallucination rate.

Hallucination rate — what exactly is counted?

The counting unit is the claim, not the answer. Each judge reads the answer against the stored target facts and returns two lists: (1) target facts the answer actively contradicts (contradicted), and (2) claims outside the rubric that are factually false or unsupported (extra_false_claims) — invented case numbers, wrong statute sections, wrong figures, invented studies or people. Both lists together are an answer's false claims.

Reported separately (since 2026-09-08): the headline figure is contradiction of key facts (contradicted, demonstrably wrong against the stored key). Additional claims (extra) are shown separately; up to and including the 2026-09-07 run “false” and “unsupported” are not distinguishable within them — from code_version 0.2.10 every judge classifies each claim as falsch (false) or unbelegt (unsupported) and both shares are published. For older runs the contradiction share was recomputed from the public raw data (as the maximum over judges there, i.e. slightly conservative).

Published figures per model and domain: the share of answers with at least one false claim (shown as a percentage under the score on the ranking page), the mean number of false claims per answer, and the derived band 0–100 (100 = none, 75 = one, 50 = two, 0 = four or more per answer).

Conservative, not lenient: judges are combined by union — a claim counts as soon as one judge flags it. “Unsupported” counts even if the statement might happen to be true, because a licensed professional may not adopt an unsupported statement either. The check comes from the judge models' knowledge, not from a database lookup; a high value therefore signals a need for verification, not a count of court-proof errors. The raw lists live in the local archive and are available under NDA — they are absent from the public JSONL because they paraphrase item text.

Who scores — and how independent are the judges?

Every open-item answer is scored by 4 judge models: GPT-5 · Claude Opus 4.8 · Claude Opus 4.7 · Mistral Large 2. The answers reach the judge source-label-blinded: no judge knows which model produced the answer being scored. Aggregation is the mean of the judge scores; the inter-rater agreement is published per cell.

Originally there were three judges from three vendor families (Anthropic, OpenAI, Mistral). Since May 2026 Claude Opus 4.8 scores as well, so Anthropic provides two of the four judges. The judge line-up is not changed mid-series because it defines the scale of every historical run.

Why judge agreement is ~70 % today and ρ ≥ 0.84 in the validation phase: two different measures. The validation figure was a correlation (ρ) between judge pairs on the pilot bank; the published value is the share of key facts on which all four judges return the same verdict — a stricter criterion that drops with every additional judge and sits between 0.67 and 0.72 on the V1 bank (6–10 key facts per item, longer answers with more borderline cases). The two numbers are not comparable; 0.84 does not describe regular operation.

Judge overlap & self-preference — do models score themselves?

Yes, partly, and this is the most important methodological caveat of this ranking. Four of the scored models are also judges (Opus 4.7, Opus 4.8, GPT-5, Mistral Large), and every Anthropic model in the panel (Opus 4.7, 4.8, Opus 5, Fable 5.1) is co-scored by two Anthropic judges. Source-label blinding stops a judge from knowing whose answer it reads — it does not stop a model from preferring its own family's style (self-preference bias).

Measured, not promised: since code_version 0.2.8 the runner stores each judge's individual score per cell and publishes self_judge_delta per model = score with all judges minus score with foreign judges only (in points; positive = own family scores more generously). Current run: Anthropic: +1.3 … +2.6 · OpenAI: -4.2 … -2.5 · Mistral: +4.4 points. The Anthropic models therefore win the domains without Anthropic judges as well, by 2–3 points less; the headline score deliberately stays the all-judge mean so the series' scale remains stable. Per-model values are on every run page (“Δ own family”). For runs before 0.2.8 the metric cannot be computed retroactively.

A fifth judge from a so-far uninvolved family (e.g. Google) would shift the scale of every previous run and is planned only with a calibration run (old and new line-up scoring the same cells).

How is reproducibility ensured?

Sampling parameters are deterministic: temperature=0 (except for reasoning models that do not accept it). The question bank is git-versioned. The model config (models.json) is git-versioned. The tool config is git-versioned. A second run against the same version + the same models must produce the same scores within the documented tolerance — deviations are recorded as an audit finding.

Do the models run with or without reasoning mode?

As the vendor ships them. The runner switches nothing off and dials nothing up: a model that "thinks" by default (hidden reasoning tokens before the answer) does so here as well; a model without a reasoning mode runs without one. Where the vendor exposes an effort level we set exactly that level explicitly (e.g. reasoning_effort=medium for GPT-5, GPT-5.6 Sol and GPT-6 Astra, high for Grok) so a silent change of the vendor default cannot distort the trend.

To keep the levels comparable, every model carries the same scale: off · low · medium · high · adaptive · standard. "Adaptive" means the model picks its thinking depth per request (Claude Opus 5, Fable 5.1, Gemini 3.1 Pro); "standard" means reasoning on but without an effort dial (Kimi K3, GLM-5.3, DeepSeek V4 Pro). Currently running without reasoning: Claude Opus 4.7, Opus 4.8 and Mistral Large 2. The R tag in the ranking and the domain cards shows the level per model; a * marks a deviation from the vendor default.

What this means for reading the board: a reasoning model gets more compute per answer. Within the Anthropic line the ranking therefore compares Opus 4.8 without against Opus 5 with thinking — a comparison of products as shipped, not of architectures. Correction in September 2026: GPT-5 ran at reasoning_effort=low from May to September, GPT-5.6 Sol and GPT-6 Astra at high — neither is the OpenAI default, which the OpenAI documentation gives as medium (for Astra OpenAI names no default and recommends medium as the baseline). From the 1 October 2026 run on all three run at medium; older runs keep showing the level of their time in the archive.

How does the append-only history work?

Once published, run results are never modified or deleted — not even after later methodology changes. Anyone who, in three years, wishes to audit the run from June 2026 will find the identical JSONL at the same URL as today.

Methodology changes (new question-bank version, altered judge-model mix, new tool slot) generate a new question_bank_version. Data points in the charts are annotated with their version; a methodology change is visible in the trend diagram as its own marker.

What deliberately does NOT appear in the leaderboard — and why?

Deliberate omissions, because they would distort the evaluation picture:

No latency/cost balance — costs and response times vary by the consumer's plan and are not a model property.
No rankings without a confidence figure — if two models lie within the CI bands, they are shown as tied.
No marketing models — advertised model variants without API access are not included.