The burden of proof is on the AI
Plausible isn't the same as proved. OpenGATE inverts the default: an answer isn't trusted until it demonstrates, mechanically, that it's grounded in the evidence it was given.
AI answer + evidence
Your system produces an answer along with the source context it should be grounded in.
Deterministic checks
Required facts present, numbers traceable, passages verbatim, abstention honoured. Pure logic — no grader model.
Regression gate
Every run is diffed against a saved baseline. Any metric that drops fails the build in CI.
Verified
Versioned scorecards make every number reproducible and auditable — ship with proof.
No LLM judge anywhere in the loop. Your judgment lives in hand-labelled gold cases, not in a grader model's mood — so verdicts cost nothing to re-run and never drift.
Four systems, one standard — and real bugs caught
OpenGATE isn't hypothetical: four PharmaTools products run on it in CI, across four different capability shapes. It has surfaced silent parse failures, driven a production model change on numbers rather than reputation (and reverted it when a pre-registered five-run replication didn't reproduce the difference), flagged an author-array inconsistency in PubCrawl's Europe PMC parser before it could reach a citation, and caught a simplifier dropping an antibiotic dose from a discharge summary — before a patient ever saw it. The newest catch came on day one of automation: the weekly gate watching that same registered medical device's live service flagged real output drift — an eval that's still watching, not one that ran once.
A silent failure mode found by the harness, eliminated with enforced structured output.
A pre-registered re-run retired the eval's own "halved hallucination" headline and reverted RefCheckr's verdict model — the gate applied to itself.
On 25 gold identifiers — two real engine bugs found, fixed, and verified closed.
Committed baseline: 0.93 recall, zero non-verbatim claims — currently red on 2 leaked non-claims.
Full methodology, model comparisons across accuracy, hallucination, latency, and cost: how RefCheckr is evaluated.
One line to verify.
The same deterministic verification, wherever your stack lives. The offline suite needs no API key.
Framework-agnostic by architecture
OpenGATE doesn't integrate with your framework — it doesn't need to. Scorers reach the system under test through a one-file adapter, and REST-backed systems need no code at all: a JSON config maps the transport. It's model-agnostic for the same reason — it verifies the system's answers against the evidence, and doesn't care whether GPT, Claude, Gemini, or an open-weights model produced them.
Worked example: a NIM-powered RAG agent, its answers gated deterministically in CI — the full adapter is one file.
Ten metric families, matched to what your system does
From turnkey answer grounding for RAG and field-level extraction accuracy to specialised checks for citation-heavy, redaction, and simplification systems.
Use both — evals measure, OpenGATE verifies
General-purpose frameworks like DeepEval and OpenAI Evals evaluate AI systems broadly, usually with an LLM judging the output. OpenGATE specialises in the narrower, harder promise: that every answer is grounded in evidence. Provenance is first-class — does the cited passage actually exist, verbatim, in the source? — and because the checks are deterministic, they're reproducible, free to run on every commit, and immune to grader drift.
Pair a general framework for broad quality metrics with OpenGATE as the gate: no prompt, model, or pipeline change ships without proving it didn't make the system less trustworthy. The approach is written up in a paper — an experience report across four production systems, with a controlled comparison against LLM-as-judge faithfulness metrics.
Open source
The whole framework is free and open — because if an evaluation influences deployment decisions, engineers should be able to inspect every scorer, metric, and benchmark.
- ✓ MIT licence — no black boxes
- ✓ npm, PyPI, and Docker packages
- ✓ CLI, GitHub Action, and MCP server
- ✓ Hand-labelled gold set included
- ✓ Contributions welcome — cases, adapters, scorers
Runs inside your infrastructure
Nothing phones home. OpenGATE is self-hosted by design — your answers, evidence, and gold cases never leave your environment.
- ✓ CPU-only Docker image, air-gap friendly
- ✓ Deterministic suite needs no API key
- ✓ Versioned, auditable scorecards (JSON + HTML)
- ✓ Per-adapter baselines for every system you gate
- ✓ Your gold set stays in your own repo
AI should be powerful. And provable.
Read the scorers, run the suite in sixty seconds, or start with the guide — the deterministic checks are free to run, forever.
OpenGATE is built by PharmaTools.AI and released under the MIT licence on GitHub. Questions about the framework or methodology: support@pharmatools.ai. See it in production on the RefCheckr evaluation page.