Not just plausible — provable

The question that matters

Can the model prove its answer from the source material?

Most AI evaluation asks whether a model produces a plausible answer. For a tool that medical writers rely on to verify clinical claims, plausibility is not enough. The failure modes that matter are subtle: a claim marked not supported because the supporting text was truncated; a slightly reworded claim verified in place of the original; a prompt change that quietly lowers verdict accuracy. None of these are visible without measurement.

OpenGATE powers the evaluation infrastructure behind RefCheckr, turning those failure modes into numbers we can track over time.

The loop in action

Before the methodology, the result. This isn't hypothetical: run against the gold set, the eval harness immediately surfaced real reliability issues and then measured the fixes — the evaluation loop doing exactly what it is for.

• It found a silent failure mode. The batch verifier was discarding any model response that didn't parse as strict JSON and defaulting it to "Not Supported." The harness measured this firing on roughly half of all multi-claim verdicts.
• It proved the fix. Switching the verdict call to enforced structured output took those parse failures from about half of cases to zero — a reliability gain on every multi-claim verification, not just in the benchmark.
• It drove a production change. Comparing model tiers on the same gold set — three repeats each, averaged — showed sonar-pro roughly halves passage hallucination (5.8% → 2.4%) against the previous production model, for meaningfully higher verdict accuracy. RefCheckr's verdict step now runs on sonar-pro as a result — a model decision made on measured evidence, not reputation.

Where it stands today: claim extraction scores about 0.95 F1 with near-full recall and a low rate of non-verbatim claims; citation detection is 100% on supported styles; and on the clearest verdicts — strong support and direct contradiction — the model is at its most reliable. Fine-grained accuracy across all six verdict types is an evolving baseline, and swings several points run-to-run, as the gold set grows toward human-labelled, real-world references.

Claim-extraction F1 and recall vary run-to-run (roughly 0.94–0.98 / 96–100%); shown as representative figures. Citation detection is 100% on supported styles — numbered and author-year (Harvard), the latter a former known gap closed via the eval loop. Verdict accuracy across all six categories is an evolving baseline, so it isn't charted as a headline figure.

The sections below explain how those numbers are produced: what the harness measures, how it works, and what it costs to run.

What it measures

The harness runs the verification pipeline against benchmark datasets of claims and supporting documents, then scores it on five families of metric:

How it works

The framework is built like a test suite for an AI system — gold cases in, metrics out, a baseline to compare against.

Choosing a model on evidence: accuracy, hallucination, latency, cost

A verifier that's accurate but hallucinates its evidence, or accurate but too slow and costly at manuscript scale, doesn't ship. A single manuscript can carry hundreds of claims, and each one is a verdict call — so every axis compounds. Rather than default to the flashiest model, we ran the harness against three Perplexity Sonar tiers on the same gold set and let the numbers decide.

Every figure below is measured — 60 verdict pairs per run, three repeats each, two to three full runs per tier, averaged (single runs wobble ±several points, so we don't trust any one). Latency and token cost are captured per call directly from the live pipeline. In each chart, teal is best on that axis and red is worst — no model wins everywhere.

Verdict accuracy — exact match on the six-point support scale  ·  higher is better

Passage hallucination — cited evidence not found verbatim in the source  ·  lower is better, and decisive for an evidence tool

Latency — p50 per claim  ·  lower is better  (scale 0–14s)

Cost — measured tokens × Perplexity rates + request fee, per 1,000 claims  ·  lower is better

Averages of two-to-three full 3-repeat runs per tier; consistency (within-pair stability) sits at 90–96% across tiers. Cost is real, not estimated: measured token usage per call — ~1,484 input / 256–307 output tokens — times Perplexity's published rates (sonar $1/$1, sonar-pro $3/$15, reasoning-pro $2/$8 per MTok) plus the default per-request fee ($5–$6 / 1,000). Reasoning-pro reports no separate reasoning tokens (its total equals input + output), so its cost is genuinely below Sonar Pro's despite the far higher latency — the reasoning happens server-side and isn't billed by token.

Why Sonar Pro is the pick — and why it isn't obvious. Sonar Pro is neither the cheapest tier nor the most accurate; it's the most expensive, and sonar-reasoning-pro labels verdicts markedly better (83% vs 62% exact). The deciding axis is passage hallucination — whether the quoted evidence actually exists in the source. For a tool whose entire promise is provable support, invented evidence is the worst failure there is, and here Sonar Pro is decisively cleanest at 2.4%, against 5.8% for sonar and 15.1% for the reasoning model. The reasoning tier's accuracy edge doesn't survive that: a verdict propped up by a passage that isn't in the source is exactly what RefCheckr exists to prevent — and it's ~3× slower on top.

What the eval changed. RefCheckr previously ran sonar in production. On the strength of these numbers its verdict step now runs on sonar-pro — same pipeline, meaningfully higher accuracy, and less than half the hallucination, accepting a higher per-claim cost as the price of evidence you can trust. That's the framework doing its job: turning a model choice into a decision backed by numbers rather than reputation, including numbers that complicate the easy answer.

Reproducibility. Each tier was run via REFCHECKR_VERDICT_MODEL=<tier> npm run eval:online; every scorecard in evals/results/ is stamped with its run_model, code SHA, p50/p95 latency, token usage and full confusion matrix. Gold set is 25 hand-labelled cases / 60 verdict pairs and growing, and single-run accuracy carries ±several points of noise — so the ranking across tiers, not any absolute number, is the durable signal.

Honest by construction

• Known gaps are tracked, not hidden. Where the system doesn't yet handle a case, the harness reports it as a named target rather than letting it pass silently — and tracked gaps get closed: author-year citations were a named gap until detection and mapping shipped, verified by the same fixture that had flagged them. The current tracked target is number-adjacent superscript citations ("week 24.1"), genuinely ambiguous with decimals. known gap
• Fidelity is a first-class metric. It isn't enough for a verdict to look right; the framework checks that quoted evidence actually exists in the source and that extracted claims are verbatim from the manuscript.
• Reproducible. Every scorecard records the code version it ran against, so a result can always be traced back to a specific state of the system.
• Growing benchmark. The gold set is expanding across citation styles and all six verdict types; metrics become more meaningful as coverage grows.

The framework behind these numbers

Teams shipping AI need repeatable, quantitative ways to compare models, validate improvements, and ensure systems stay trustworthy over time. For evidence-grounded AI — systems that must ground answers in source documents — that measurement layer is still largely missing.

Everything on this page is produced by OpenGATE (Open Grounded AI Testing & Evaluation), an open-source framework for evaluating evidence-grounded AI systems. Although originally developed for RefCheckr, its structure — gold sets, scorers for citation and factual support, fidelity and hallucination checks, and a regression gate — can evaluate any AI workflow that has to prove its answers, from RAG and document QA to medical, legal, and regulatory AI.

First implementation of OpenGATE. Not the last.

RefCheckr proved the framework in production, but OpenGATE is intentionally system-agnostic. Any AI that grounds its outputs in source documents — whether in healthcare, legal, scientific publishing, or enterprise search — can adopt the same evaluation pattern: benchmark datasets, automated scorers, reproducible metrics, and regression gates. The methodology travels; only the gold set changes. Read about the framework →