How RefCheckr verifies a claim

Core principle

RefCheckr is built on one rule: a verdict is only as strong as the verbatim quote that supports it. The model is asked to find evidence in the reference, not to reason about the topic. If it can't quote the exact wording from the document, the strongest verdict it can return is "implied by data" — never "strong support."

This makes RefCheckr deliberately conservative. It will not "fill in" support that isn't in the text, even when it would be biomedically reasonable to do so.

The verification pipeline

When a user verifies a claim against a reference PDF, the request flows through these steps:

Models in use

Each model is pinned by exact version ID. Updates are tested in a non-production environment before deploy. None of these providers train on RefCheckr customer content.

Anti-hallucination guardrails

• Verbatim quote rule — every cited passage must include the first and last ~15 words of the original verbatim. The system prompt explicitly forbids paraphrasing or reconstructing quotes.
• Server-side quote validation — for every "strong support" verdict, the server checks that at least one cited passage can actually be matched in the original PDF text. Hallucinated quotes get the verdict downgraded automatically.
• Page-level traceability — every passage carries a page number extracted from [Page X] markers, so a reviewer can audit the verdict against a specific page of the source document.
• Strict verdict definitions — "strong support" requires explicit narrative text. Tables, figures, or numbers alone produce "implied by data" instead.
• Abstract exclusion — the abstract section of source papers is deliberately stripped before analysis. Pharma medical writing requires support from the body of the paper, not the abstract.

Find candidate references — two-pass validation

When a claim is not supported by the user's reference, RefCheckr can search PubMed for alternative references that may support it. Candidate references are not just retrieved — they are checked against the claim before being shown.

1. Broad PubMed retrieval — an unfiltered relevance-sorted search returns up to 15 candidates. We deliberately do not pre-filter by publication type at search time, since strong evidence sometimes lives in non-RCT-tagged papers.
2. Local re-rank — candidates are sorted by a publication-type score (meta-analyses, systematic reviews, RCTs, and guidelines rank above editorials and commentary), then by PubMed's relevance order as tie-break. The top 8 advance.
3. Pass 1 — Sonar triage — a single Perplexity call assesses all 8 abstracts and labels each as direct, partial, or none. Candidates labelled none are dropped.
4. Pass 2 — Claude validation — the top 3 surviving candidates are sent in parallel to Claude Haiku for a per-candidate verdict, confidence score, verbatim relevant snippet, and rationale.
5. Fallback — if Claude validation fails for a candidate, the Sonar Pass 1 verdict is used and tagged validated_by: "sonar" rather than dropping the candidate. The frontend and server logs record which model produced the final verdict.
6. Output — only candidates assessed as "directly supports" or "partially supports" are returned. Cautious wording is used throughout: candidates are presented as "for review", not as endorsed citations.

Error handling

• Per-call timeouts — verify calls timeout at 90 seconds, triage and candidate validation at 60. A timeout returns a structured error rather than a stale or partial verdict.
• Malformed model output — if a model returns non-parseable JSON, the server falls back to a "not supported / unable to analyze" verdict rather than guessing.
• Provider unavailability — Claude validation failures in find-supporting fall back to Sonar's triage verdict; visual analysis failures simply omit the visual layer rather than failing the request. Each request logs which fallback path was taken.
• Rate limits & quotas — API requests are limited to 30 per hour per IP. Per-user monthly quotas (free / pro tier) are enforced server-side.
• Diagnostic logging — for find-supporting, the post-rerank candidate list and per-PMID Sonar verdicts are always logged, so a zero-result outcome can be diagnosed (retrieval failure vs validation rejection) from server logs alone.

Model lifecycle

Models are pinned to specific versions in source control (e.g. claude-haiku-4-5-20251001, claude-sonnet-4-20250514). New model versions are introduced deliberately rather than automatically — any change in model behaviour can be traced to a specific deploy in version history.

Customer content is never used for model training. Anthropic's and Perplexity's API terms prevent providers from training on customer-submitted content by default, and RefCheckr does not perform any fine-tuning or retrieval-augmented training on user data.

Known limitations

• English-only — RefCheckr is trained against English biomedical literature. Other languages may work but accuracy is not validated; multi-language support is on the roadmap.
• Heavily visual decks — for slide decks where the message lives in figures rather than narrative, the visual analysis path is less reliable than the textual path.
• Reference scope — verdicts reflect the evidence in the reference provided, not the broader biomedical literature. RefCheckr will not "vouch for" a claim using prior knowledge of the topic.
• OCR quality — for scanned references, OCR errors can introduce minor character-level differences that affect verbatim quote matching.
• Conservative bias — by design, RefCheckr will downgrade a verdict to "implied by data" when narrative support is absent. Reviewers should expect more partial verdicts than a less strict tool would return.