The whole model, in four steps
You don't hand OpenGATE your model. You describe what a good answer looks like (gold cases), point it at your system's interface, and it does the rest — on every commit.
Which one matches your system?
| Your system… | Capability | What gets scored |
|---|---|---|
| Answers questions from retrieved context — RAG, document QA, legal AI, scientific assistants turnkey | grounding |
Did it answer correctly from context, without inventing facts, and abstain when the context lacks the answer? |
| Extracts claims and verifies them against cited references on a graded scale | qa | Claim extraction quality, verdict accuracy, passage hallucination |
| Rewrites text for a different audience — simplification, translation, summarisation | simplify | Do critical facts survive the rewrite, with nothing fabricated? |
| Removes identifiers / PII from text | redaction | Are all gold identifiers actually removed? |
| Retrieves records from an authority — search / database wrappers | retrieval | Does the record match the source — no dropped or garbled fields? |
| Fills a structured record (JSON Schema) from a document — referrals, applications, forms | extraction | Schema validity, per-field accuracy vs hand-labelled gold, no fabricated fields — unknown → null, never guessed |
Most RAG and QA teams want grounding. It's the turnkey path — no verdict taxonomy to adopt, no citation-mapping contract. The rest of this guide follows it. (The qa capability is richer but is shaped around RefCheckr's six-point support scale; if that fits you, see the adapter contract on GitHub.)
No code, or fifteen lines
If your system has an HTTP endpoint, you write zero code — just a config file pointing at it. Placeholders like ${MY_RAG_URL} read from the environment, so tokens never sit in the file.
Your /api/answer receives { question, context } and returns { "text": "…" }. That's it.
Where your domain knowledge goes
A grounding case is a question, the context your retriever would return, and the facts a correct answer must contain — asserted as anchors with alias variants, not a verbatim expected answer. Paraphrase is fine; dropping a fact is not.
Then add the case that catches hallucination — one where the context doesn't hold the answer. A faithful system abstains; a confident fabricator gets caught:
Start with a handful covering your real failure modes; coverage grows over time. The package ships runnable examples — a SaaS refund policy, a legal termination clause, an unanswerable case — so you can copy one and edit.
Run it, save a baseline, fail the build on regressions
Run against your system. Every failure is named — "missing answer fact …", "ungrounded number …", "did not abstain" — so you know exactly what broke.
Add --report and you also get a self-contained HTML dashboard — pass/fail per scorer, deltas against your baseline, and every named failure, in one file you can open or attach to a CI run. No server, no dependencies:
| answer_recall | 90.0% | ▼ -5.0pp |
| ungrounded_numbers | 2 | |
| abstention_rate | 100.0% | ±0 |
| fidelity | 100.0% | ±0 |
A real --report dashboard: verdict, per-scorer metrics, baseline deltas, and named failures.
Save a baseline once you're happy (baselines are per-adapter, so different systems don't collide), commit it, and drop in the GitHub Action:
Now any change that drops answer recall, starts inventing figures, or stops abstaining fails the build before it ships.
What OpenGATE deliberately won't do
- No LLM-as-judge. Scores are deterministic checks against hand-labelled gold — reproducible, free to run in CI, and your judgment lives in the gold set, not a grader model's.
- No general "quality" score. It measures whether an answer is grounded in its evidence, not whether it's fluent or helpful. Pair it with a general framework like DeepEval for breadth; use OpenGATE to gate the grounding.
- It never sees your model. It calls your system's interface, exactly as a user would — no weights, no internals.
Three ways to run the same check
The grounding check is one piece of deterministic logic. Use it in whichever surface fits your stack — the verdict is identical.
Gate your whole system
Adapter or HTTP config, gold cases, a baseline — fail the build when grounding slips. This guide. Also a CPU-only Docker image.
Agents self-check inline
One tool, check_grounding, so an agent verifies its own answer before replying. Local; npm or desktop extension.
Drop into pytest or DeepEval
Zero-dependency. check_grounding() inline, assert_grounded() in tests, or a DeepEval GroundingMetric.
Worked example: evaluate an NVIDIA NIM–powered RAG agent — the same grounding gate on a real agent's answers, with a runnable notebook and a Node CI adapter.
Point it at your system
The full written guide, every capability's contract, and the gold-case reference are on GitHub. Installable from npm.
Prefer the terminal? npx @pharmatools/opengate runs the offline suite in seconds. Questions, or a system that doesn't fit a capability? Open an issue. See also the OpenGATE overview.