Evaluation for AI that has to prove its answers

Not a benchmark. Infrastructure.

OpenGATE is not another generic LLM benchmark. It is an engineering framework for continuously measuring and improving evidence-grounded AI systems: benchmark datasets, regression testing, prompt evaluation, model comparison, pipeline comparison, CI/CD quality gates, and quantitative metrics. The goal is that every model, prompt, or workflow change can be automatically evaluated before deployment — the same discipline a test suite brings to conventional software.

That inverts the usual shape. Most portfolios are a set of independent apps. This one is shared evaluation infrastructure with applications built on top — one framework, held to one standard, underneath every product:

Philosophy

Can the model prove its answer from the source material?

Modern AI evaluation often measures whether an answer sounds plausible. OpenGATE instead measures whether an answer is faithfully grounded in evidence. The failure modes that matter in production are subtle: a supporting passage that doesn't exist in the source; a slightly reworded claim verified in place of the original; a prompt change that quietly lowers accuracy. None of these are visible without measurement.

Evidence matters more than plausibility. That distinction is decisive wherever an ungrounded answer carries real cost:

What it measures

OpenGATE runs a system's pipeline against benchmark datasets, then scores it on a growing set of metric families — matched to what a system does, from claim verification to answer grounding:

How it works

The framework is built like a test suite for an AI system — gold cases in, metrics out, a baseline to compare against. Where it sits in the development loop:

One diagram, the whole framework: no change ships without proving it didn't make the system less reliable.

Integration is deliberately thin. Scorers reach the system under test through a pluggable adapter — one file with a couple of exports, validated at load — and REST-backed systems need no code at all: a JSON-configured HTTP adapter maps the transport. A GitHub Action wraps the regression gate, so any repository can keep its own gold set and baseline in its own tree and fail the build when grounding slips.

Evaluate your own system

OpenGATE isn't only for the applications below — it's built for any team with a RAG pipeline, document-QA tool, legal assistant, or scientific-literature system that has to ground its answers. It's the same deterministic check — required facts present, every number traceable, abstention when the context can't answer — in whichever surface fits your stack. No model access, no LLM judge.

Worked example: evaluate an NVIDIA NIM–powered RAG agent — the same deterministic grounding check gating a real agent's answers in CI, framework- and model-agnostic.

First implementation: RefCheckr

OpenGATE powers the evaluation infrastructure behind RefCheckr, an evidence verification tool for medical writers. The framework isn't hypothetical — run against RefCheckr's gold set, it surfaced real reliability issues, measured the fixes, and drove a production model change on evidence rather than reputation.

Full methodology, model comparisons across accuracy, hallucination, latency, and cost, and reproducibility notes are on the RefCheckr evaluation page.

Second implementation: Redacta

Redacta exercises the framework's redaction capability — proof the methodology isn't QA-shaped. On its first run against a new gold set of synthetic UK clinical notes, the eval found two real bugs in the shipped engine: relation phrases like "Next of kin:" swallowed nested name matches, and apostrophe surnames (O'Brien) were dropped from name capture. Both were fixed the same day, verified by the eval's known-gap tracking flipping to closed, and shipped across every Redacta surface — npm, MCP server, CLI, Python, and the iPhone app. Street-address detection followed, closing the last tracked gap. Current scorecard: 100% recall on 25 gold identifiers, 0 leaks, no open gaps — every gap the eval has found has since been closed.

Third implementation: Patiently AI

Patiently AI exercises the simplify capability — faithfulness for text that is paraphrase by design, scored on whether safety-critical specifics survive. Its first production run caught the simplifier dropping an antibiotic dose from a discharge summary and a lab value from a results letter. A preservation rule in the prompts fixed it the same day, verified by the eval: 100% anchor recall, zero dropped facts, zero fabricated numbers — with readability slightly improved.

Fourth implementation: PubCrawl

PubCrawl is the odd one out — an MCP server wrapping PubMed and ClinicalTrials.gov, with no model. It exercises the retrieval capability: the deterministic layer everything else grounds on. A silent XML-parser regression — a collapsed author array, a truncated abstract — would poison every citation built on the record, so the eval checks retrieval fidelity against hand-verified anchors, driving PubCrawl through its real interface so the full production parse path is tested. First run: 100% fidelity, every field intact. That OpenGATE scores a non-AI system at all is the point — evidence-grounded AI is only as trustworthy as the retrieval beneath it, so the retrieval belongs in the same regression gate.

One framework, every system

Four PharmaTools products already run on it — evidence verification, redaction, simplification, retrieval — four different capability shapes, one evaluation standard. And nothing about it is PharmaTools-specific: the structure — gold sets, per-metric scorers, fidelity and hallucination checks, reproducible scorecards, and a regression gate — generalises to any system that has to prove its answers, from retrieval-augmented generation and document QA to legal, regulatory, and scientific assistants. Point it at your own with one command (opengate init).

Within the PharmaTools AI ecosystem, OpenGATE sits alongside PubCrawl as shared infrastructure: PubCrawl handles biomedical retrieval, OpenGATE handles evaluation, and applications like RefCheckr and Patiently AI build on both.

How it relates to existing tools: general-purpose frameworks such as DeepEval and OpenAI Evals evaluate AI systems in general. OpenGATE specialises in systems that must justify every answer from underlying evidence — provenance, citation fidelity, verbatim passage checks, and regression detection are first-class concerns rather than plugins. The two are complementary: you might use a general framework for broad quality metrics and OpenGATE to gate the grounding.

Why open source?

Evaluation frameworks shouldn't be black boxes. If an evaluation influences deployment decisions, engineers should be able to inspect every scorer, metric, and benchmark — read the code, rerun a scorecard, and audit exactly how a number was produced. Open-sourcing OpenGATE is also a bet on community: transparency and reproducibility make results trustworthy, and shared gold sets and adapters make the framework more useful with every implementation. It ships with a reference adapter for RefCheckr, a config-driven HTTP adapter for REST-backed systems, a GitHub Action for CI, and a hand-labelled biomedical gold set that doubles as a worked example of the case format — published as @pharmatools/opengate on npm, so trying it is one npx command.