Name: OpenGATE
Author: PharmaTools.AI

How it works

The burden of proof is on the AI

Plausible isn't the same as proved. OpenGATE inverts the default: an answer isn't trusted until it demonstrates, mechanically, that it's grounded in the evidence it was given.

◇

AI answer + evidence

Your system produces an answer along with the source context it should be grounded in.

→

⚙

Deterministic checks

Required facts present, numbers traceable, passages verbatim, abstention honoured. Pure logic — no grader model.

→

⇅

Regression gate

Every run is diffed against a saved baseline. Any metric that drops fails the build in CI.

→

✓

Verified

Versioned scorecards make every number reproducible and auditable — ship with proof.

No LLM judge anywhere in the loop. Your judgment lives in hand-labelled gold cases, not in a grader model's mood — so verdicts cost nothing to re-run and never drift.

Proven in production

Four systems, one standard — and real bugs caught

OpenGATE isn't hypothetical: four PharmaTools products run on it in CI, across four different capability shapes. It has surfaced silent parse failures, driven a production model change on numbers rather than reputation, flagged an author-array inconsistency in PubCrawl's Europe PMC parser before it could reach a citation, and caught a simplifier dropping an antibiotic dose from a discharge summary — before a patient ever saw it.

Parse failures

~50% → 0

A silent failure mode found by the harness, eliminated with enforced structured output.

Passage hallucination

5.8% → 2.4%

Cut by more than half — the eval drove RefCheckr's production model switch.

Redaction recall

100% · 0 leaks

On 25 gold identifiers — two real engine bugs found, fixed, and verified closed.

Claim extraction

~0.95 F1

Near-full recall on the gold set, with a low rate of non-verbatim claims.

RefCheckrEvidence verification — citations, claims, verdicts RedactaRedaction — every leak a named failure Patiently AISimplification — do critical facts survive paraphrase? PubCrawlRetrieval fidelity — caught a Europe PMC parser slip, held at 1.0

Full methodology, model comparisons across accuracy, hallucination, latency, and cost: how RefCheckr is evaluated.

One line to verify.

The same deterministic verification, wherever your stack lives. The offline suite needs no API key.

# run the evaluation suite — no install, no API key $ npx @pharmatools/opengate OpenGATE — 39 case(s), online=false, adapter=refcheckr ✓ citation-detection PASS # scaffold gold cases + HTTP config + a GitHub Action for your system $ npx @pharmatools/opengate init

# pip install opengate-grounding — zero dependencies from opengate_grounding import check_grounding result = check_grounding( answer, context, # the retrieved evidence anchors=["Reye's syndrome"], # facts the answer must contain ) result.grounded # True — deterministic, every run # or gate a test: assert_grounded(answer, context, ...) · DeepEval: GroundingMetric

# MCP server — agents verify their own answers before replying $ npx @pharmatools/opengate-mcp # exposes one tool: check_grounding # runs locally, via npm or as a desktop extension

# .github/workflows/verify.yml — drop-in CI gate for any repo - uses: nickjlamb/opengate@v0 # keeps your gold set and baseline in your own tree; # the build fails when grounding slips

# CPU-only image for containerised pipelines $ docker run pharmatools/opengate

Getting started guide → npm → PyPI → Docker Hub →

Works with your stack

Framework-agnostic by architecture

OpenGATE doesn't integrate with your framework — it doesn't need to. Scorers reach the system under test through a one-file adapter, and REST-backed systems need no code at all: a JSON config maps the transport. It's model-agnostic for the same reason — it verifies the system's answers against the evidence, and doesn't care whether GPT, Claude, Gemini, or an open-weights model produced them.

Python pip · pytest Node.js npm · npx DeepEval GroundingMetric GitHub Actions CI gate Docker CPU-only image MCP agent self-checks REST / HTTP no-code adapter NVIDIA NIM worked example

Worked example: a NIM-powered RAG agent, its answers gated deterministically in CI — the full adapter is one file.

What it checks

Ten metric families, matched to what your system does

From turnkey answer grounding for RAG and field-level extraction accuracy to specialised checks for citation-heavy, redaction, and simplification systems.

Answer groundingThe turnkey path for RAG and document QA: correct facts, no invented figures, abstention when the context can't answer.

Hallucination rateAre supporting passages genuinely present in the source, verbatim? Anything that fails counts as an infidelity.

Citation accuracyThe right citation markers, mapped to the right references — scored by exact set-match and Jaccard overlap.

Claim extractionThe genuinely verifiable claims, and nothing else — precision, recall, and F1 against gold.

Factual supportDoes each claim, checked against its cited source, get the correct verdict on a graded scale?

ConsistencySame input, repeated runs — does the verdict hold? Non-determinism is quantified, not assumed away.

Redaction recallGold-labelled identifiers actually removed, every leak a named failure — over-redaction counted too.

Simplification faithfulnessParaphrase by design: do doses, values, and timeframes survive, with every number traceable?

Retrieval fidelityDoes the record a client receives match the authority? Catches parser regressions before they poison citations.

Extraction accuracyStructured extraction against hand-labelled gold, per field: schema validity, normalised value accuracy, and fabricated fields named — a value where the document states nothing.

Verification vs evaluation

Use both — evals measure, OpenGATE verifies

General-purpose frameworks like DeepEval and OpenAI Evals evaluate AI systems broadly, usually with an LLM judging the output. OpenGATE specialises in the narrower, harder promise: that every answer is grounded in evidence. Provenance is first-class — does the cited passage actually exist, verbatim, in the source? — and because the checks are deterministic, they're reproducible, free to run on every commit, and immune to grader drift.

The methodology travels; only the gold set changes.

Pair a general framework for broad quality metrics with OpenGATE as the gate: no prompt, model, or pipeline change ships without proving it didn't make the system less trustworthy. The approach is written up in a paper — an experience report across four production systems, with a controlled comparison against LLM-as-judge faithfulness metrics.

Open source

The whole framework is free and open — because if an evaluation influences deployment decisions, engineers should be able to inspect every scorer, metric, and benchmark.

✓ MIT licence — no black boxes
✓ npm, PyPI, and Docker packages
✓ CLI, GitHub Action, and MCP server
✓ Hand-labelled gold set included
✓ Contributions welcome — cases, adapters, scorers

View on GitHub

Runs inside your infrastructure

Nothing phones home. OpenGATE is self-hosted by design — your answers, evidence, and gold cases never leave your environment.

✓ CPU-only Docker image, air-gap friendly
✓ Deterministic suite needs no API key
✓ Versioned, auditable scorecards (JSON + HTML)
✓ Per-adapter baselines for every system you gate
✓ Your gold set stays in your own repo

Talk to us

AI should be powerful. And provable.

Read the scorers, run the suite in sixty seconds, or start with the guide — the deterministic checks are free to run, forever.

Get started View on GitHub Read the paper

OpenGATE is built by PharmaTools.AI and released under the MIT licence on GitHub. Questions about the framework or methodology: support@pharmatools.ai. See it in production on the RefCheckr evaluation page.

Don't trust AI. Verify it.