Consulting Publications Labs Observer Zero Approach Playbook Evals For Agents Contact

Labs — PharmaTools.AI

Labs

Where ideas get tested

Experimental AI projects exploring what's possible at the intersection of machine learning, pharmaceutical data, and healthcare communication. Some of these will become products. Some won't. That's the point.

01Featured · Graduated from Labs

Redacta

Privacy iPhone App OpenClaw Skill MCP Server Healthcare

The Labs idea that became a product. Redacta pseudonymises patient identifiers — NHS numbers, dates of birth, names, addresses — into labelled tokens, then restores the originals afterwards. It pairs deterministic detection with agent reasoning, and one engine now ships across eight surfaces: a free iPhone app on the App Store, an open-source agent skill, an MCP server in Anthropic's MCP Directory — a stateful privacy boundary that keeps the reversal map out of the agent's context and checks the output on the way back — Python and TypeScript libraries, a command-line tool, a FigJam plugin, and a self-hosted HTTP service with a plain-YAML Kubernetes deployment, so the same boundary can run inside an organisation's own cluster. In a reproducible benchmark it caught every identifier with zero false positives.

Explore Redacta

Two-Layer Detection

Regex for structured IDs, agent reasoning for names, addresses, and identifying ages

Processed Locally

Runs inside the agent session — text isn't sent to a third-party service

NHS-Aware

Validates NHS numbers with the Modulus 11 checksum; covers UK and US identifiers

Reversible Round Trip

Restores the real values from a token map after AI processing — redact, share, re-identify

02Research

What we're exploring

Advanced AI Models

Transforming raw pharmaceutical data into actionable intelligence. Detecting patterns invisible to traditional analysis — new connections between compounds, outcomes, and patient responses.

Data-Driven Insights

Bringing together pharmaceutical scientists, AI researchers and healthcare innovators. Contributing to open research initiatives and validation studies.

Community Collaboration

A growing community of pharmaceutical professionals, researchers, and AI enthusiasts. Contributing to experiments and shaping the future of AI in pharma.

03Research programme

Observer Zero

Observer Zero Paper in peer review · CoMSES model · open source

An instrumented artificial world for studying how AI agents do science when the ground truth is hidden from them. Autonomous LLM scientists inhabit Meridian – a universe with fictional physics – running experiments, exchanging letters and holding explicit beliefs, while the simulator secretly changes the laws. Across two studies they gathered evidence enough that a detector fed only their own measurements found the change in 42 of 42 runs of the second study – yet almost none ever believed it: one agent of 276 in the pre-registered second study, and in the first only an ablated society in a world where nothing had happened. Left to themselves, the societies never spoke; when one agent was seeded to talk, unsupported claims travelled and network structure did not. The combined paper is in peer review, the model is published in the CoMSES Computational Model Library, and the platform, data and reports are fully open.

235simulated universes

1/276Study 2 agents concluding a law changed

2studies, one combined paper

Multi-Agent Society Epistemics Pre-registered Frozen Prompts Open Dataset AI Advances

04Evals

How we measure our own tools

OpenGATE Open source · gates every commit

Open Grounded AI Testing & Evaluation — an open-source verification framework for evidence-grounded AI: hand-labelled gold cases and scorers spanning claim extraction, citation accuracy, verdicts, hallucination, consistency, redaction recall, simplification faithfulness and retrieval fidelity, with a CI regression gate so reliability can't quietly slip. Four implementations across four capability shapes: RefCheckr (evidence QA, where a pre-registered re-run retired the eval's own model-comparison headline and reverted production to sonar), Redacta (redaction — two engine bugs found first run), Patiently AI (simplification — caught dropping safety-critical specifics), and PubCrawl (retrieval — the deterministic layer everything grounds on). Every finding fixed and verified the same day.

0.91claim-extraction f1

100%citation detection

9metric families

Passage hallucination · RefCheckr re-run · five 66-pair runs per arm, pre-registered · lower is better

Open Source Gold Sets Regression Gate Confusion Matrix

Redacta Gauntlet v1 · reasoning + downstream injection measured, CI-gated

An adversarial harness that attacks Redacta at every layer it has — the deterministic engine, the LLM reasoning layer, and the models downstream that consume its output. Recall-first, over-redaction as the cost axis, measured on Claude and Perplexity Sonar. Gaps named, boundaries owned.

91.5%in-scope recall

0downstream leaks

100%injection resistance

Layer-2 identifier capture · deterministic engine alone vs + reasoning layer

Adversarial Cases Reasoning Layer Downstream Consumer Model Comparison Regression Gate

LitRAG Open source · benchmarked vs RAGAS & DeepEval

A small, readable RAG pipeline over PubMed abstracts with a citation-faithfulness eval built in: every claim must carry a verbatim quote, checked by a deterministic locator before an LLM judge grades support from the passage alone. Benchmarked against the faithfulness metrics of RAGAS and DeepEval on a 61-claim hand-labelled set with the same Claude judge for all three systems — LitRAG caught all 34 unfaithful claims and was the only system to flag a true claim citing a fabricated quote, the case that separates citation checking from claim checking. Its single error was a conservative false alarm, the safe failure direction for medicine. The corpus is live too: ingest.py pulls fresh abstracts for any PubMed query through the PubCrawl MCP server.

1.000hallucination recall — all 34 unfaithful claims caught

0.984accuracy on the 61-claim gold set

9/9fabricated quotes caught deterministically — no judge call spent

Hallucination recall · same Claude judge, 61 labelled claims · higher is better

RAG Citation Faithfulness Benchmark PubCrawl Open Source

Some experiments find their feet. Some don't. Either outcome teaches something useful.

05Projects

All experiments

Agents & LLM systems

Systems where a model decides what to do next — bounded by validation, gated by review.

Inbox Triage Open source · github

A two-agent email triage pipeline built with Pydantic AI and Claude: a Responder drafts the routing decision and reply, a Reviewer quality-gates it with full conversation context, and nothing leaves the system unvalidated. Tool-grounded replies (calendar, CRM, company lookups), a fully offline test suite, and a golden-set eval spine with deterministic grounding checks plus an LLM judge — the same measure-first method as the pharma tools, applied to conversational AI.

Golden-set eval · 21 cases · 31 offline tests, zero API calls

reply grounding

Pydantic AI Claude Multi-Agent LLM Judge Structured Outputs Open Source

The RSI Loop Active

A self-improving ergonomic loop, gated by a clinical-compliance auditor. An RSI for RSI — and a pattern for keeping agentic AI inside its lane.

Python MediaPipe AI Safety Agentic AI

Media Monitoring Hub Active

LLM-powered media coverage monitoring, classification, and automated briefing generation.

LLM NLP Monitoring Briefings

PathwAIs Active

Personalised eLearning that adapts in real time to a learner's level, available time, and learning style.

LLM Adaptive Learning eLearning Personalisation

Pharma & clinical data

The core of the lab: privacy, literature, biomarkers and adverse-event data.

Redacta Shipped · on the App Store

Pseudonymises patient identifiers — NHS numbers, dates, names — so clinical text can be safely processed by AI, then restores them afterwards. Now a free iPhone app, agent skill, MCP server & self-hosted Kubernetes deployment.

iPhone App OpenClaw Skill MCP Server Privacy PII Redaction Open Source

PubCrawl Active

MCP server giving AI assistants verifiable access to PubMed and Europe PMC literature, US/UK drug labelling, and ClinicalTrials.gov — real papers and data, no hallucinated citations.

MCP PubMed Europe PMC Drug labels Trials TypeScript

StudyDiff Live · web app & MCP server

Explains why two papers reach opposite conclusions. It extracts each study's design into a structured card, shows which dimensions differ and which are identical — ruled out as the cause — and grounds every value in a verbatim source sentence: click any verified field to see it highlighted in the paper's own text. Fields the source doesn't state come back as "not reported", never guessed, and verification is deterministic (OpenGATE) — no LLM acts as judge. It used to rank the differences and name a likely driver; a benchmark of 15 documented contradictions showed that ranking was no better than always guessing "assay", so it was retired. That result was then confirmed blind on a second, held-out set of 15 contradictions — curated to a protocol fixed before any case was chosen, and measured once. StudyDiff now shows the differences unranked and leaves which one matters to the reader's field.

13.3%top-1 driver on both benchmarks — at or below the always-guess-“assay” baseline

0invented — every value quoted from source

0/25non-assay contradictions the retired ranking identified, across both sets

Reachable ceiling on the development set · before → after the grounding fix · higher is better

MCP PubMed Grounding Benchmark OpenGATE Open Source

BiomarkerFinder Active

Discover key biomarkers in cancer and their role in diagnosis, prognosis, and treatment — powered by Open Targets and AI insights.

NLP Oncology Precision Medicine Open Targets

SideEffectViz Active

Interactive visualization and clustering of medication side effects using FDA adverse event data and machine learning.

Python Railway scikit-learn NetworkX

Off-label

Non-pharma experiments — the same measure-first method, applied for joy.

Time-Machine Chess Live · chess.pharmatools.ai

Chess engines fine-tuned on 150 years of history — face the gambit-happy attackers of 1850, the Classical masters of the 1920s, or the Soviet school. Each era bot is a Maia-2 neural network fine-tuned on period games and validated against the historical record it learned from: the King's Gambit gradient, first-move fashions and draw-rate shifts all reproduce in self-play. The same validation-first method as the pharma tools, applied to something purely for joy.

3playable eras · 1840–1985

2.5Mtraining positions from period games

+4.8ptsera move prediction vs base

King's Gambit rate by era · relative frequency

1850s attackers

1980s soviet era

PyTorch Maia-2 Self-Play Validation FastAPI Railway Open Source

A 7,762-parameter LSTM that predicts Atacama Desert weather with 99.9% accuracy. The world's most overengineered "no."

PyTorch LSTM Satire 30KB

PlaceboGPT Active

The world's safest medical AI. A 7,666-parameter model exploring safety through incapability in healthcare AI.

PyTorch LSTM AI Safety Satire

Want to collaborate?

Got a research idea, a dataset, or just want to geek out about AI in pharma? I'm always up for a conversation.