Groundwork — From AI Prototype to Safe Production

The problem

The gap between "it works in the demo" and "we can put this in front of people"

Plenty of small teams have built a document-QA assistant that answers well most of the time. What they don't have is the apparatus around it: a privacy pre-step, an evaluation built from their own failures, and a gate that stops a quiet prompt tweak from shipping a system that fabricates. That apparatus usually requires an ML engineer. Groundwork scaffolds it instead.

One command writes the harness into your repo. One file connects it to your system. The checks are deterministic — same inputs, same result, every time, free.

npm install -D @pharmatools/groundwork npx groundwork init

Sixty seconds

See the gate catch fabrication before you wire anything

The repo ships a fully offline demo — a tiny document-QA system wrapped in the exact harness init scaffolds. Run it, then flip the system into the failure modes real systems actually have and watch the build turn red:

# the whole loop passes, fully offline npm run demo # the system starts inventing figures and answering when it should abstain npm run demo:break ✗ GROUNDING demo-savings: ungrounded number "14" — not in the provided context ✗ GROUNDING demo-unanswerable-appeals: unanswerable question — the answer did not abstain

That red build is the product: a fabricated figure or a failed abstention becomes a CI failure, not a user complaint.

The method

Four steps, one afternoon

Wire your system — one file

Edit groundwork/adapter.mjs: it receives { question, context } and returns { text }. HTTP endpoint or local module — the eval never touches your system except through this file.

Redact before anything leaves the machine

Every question and context runs through a deterministic redaction pass (Redacta) first — identifiers become labelled tokens locally. The same ~40 lines belong in your production path, so what you evaluate is what you ship.

Build your gold set from real failures

Every case comes from a question a user really asked or a failure you really saw. Anchors pin the facts a correct answer must contain; every number must trace to your documents; and at least one case must be unanswerable — where the only correct behaviour is to say so.

Freeze a baseline, gate your CI

groundwork check --baseline makes a good run the floor. The scaffolded GitHub Action fails any pull request that scores below it. Reliability can't quietly regress.

A case, concretely

What a gold case looks like

Plain JSON, written by the person who knows the documents — no eval expertise required:

{ "id": "max-daily", "question": "What is the maximum amount I can take in 24 hours?", "context": "…the passage your retrieval returned…", "answerAnchors": [{ "value": "100 mg", "aliases": ["100mg"] }], "allowedNewNumbers": [], "answerable": true }

The scorer checks three things, deterministically: the anchored facts appear, every number traces to the context, and unanswerable questions get a refusal instead of an invention.

Costs

Measure first, optimise second

groundwork cost reads real token usage from your eval runs and recommends savings in leverage order: prompt caching for the documents that ride along with every question, batching for anything not user-facing-urgent, context trimming (which is also safer), and model routing only after those — re-checked through the gate, because cheaper is only better if the grounding score doesn't move.

In Claude

The same gate, conversationally

Groundwork ships an MCP server, so Claude Code, Cowork, and Claude Desktop can run the checks as tools — scaffold a harness, run the full readiness gate on a repo, or paste a single question, context, and answer and get the deterministic verdict with named issues, no repo required. The conversational check imports the exact same gate logic as CI, so the two can never drift apart.

{ "mcpServers": { "groundwork": { "command": "npx", "args": ["-y", "-p", "@pharmatools/groundwork", "groundwork-mcp"] } } }

There's also an Agent Skill that teaches an agent to run the gate and report results honestly — including refusing to present a green check as a safety certification. Both ship in the npm package.

Honestly

What Groundwork is not

Groundwork is a strong floor, not a guarantee. It automates the checks that can be automated — redaction, grounding, regression gating, cost hygiene. It cannot certify an AI system safe, and it does not replace human review for answers that affect someone's health, money, legal standing, or safety. Keep a person in that loop. That is not a limitation of this tool so much as the current state of the field, stated plainly.

Under the hood

Built on tools you can inspect

OpenGATE

The evaluation engine: gold-anchored, LLM-judge-free scoring for evidence-grounded AI.

About OpenGATE →

Redacta

The privacy pre-step: deterministic, on-device pseudonymisation of identifiers before AI sees your text.

About Redacta →

A real, live example

A Claude-backed document-QA system with prompt caching, key-gated evals, and measured token usage — plus a mock API so the whole loop runs offline in CI.

examples/claude-doc-qa →

The full walkthrough

Install to gated CI in about half an hour, most of it spent writing gold cases — the half hour that matters.

Getting started →