How accurate is Redacta?

The corpus is 300 synthetic UK clinical notes with 1,713 gold-labelled identifiers, plus 1,329 "preserve" distractors designed to tempt a naive redactor. It is run against the shipping engine in clinical mode. All data is synthetic — no real patient information is used.

A number means more next to a baseline. We ran Microsoft Presidio — the most widely used open-source PII detector — over the identical corpus, scored by the identical rule (presidio-analyzer 2.2.362, default recognisers, en_core_web_lg 3.7). The gap isn't really about finding identifiers. It's about not shredding the record around them.

Presidio finds most identifiers — but it also flags 1,643 things that aren't personal data: every clinician's name, every appointment date. It keeps barely a third of the context a clinical record needs to stay readable. And it has no recogniser for UK National Insurance numbers (32% found) or postcodes (38%) — identifiers Redacta is purpose-built for. On the same text, Redacta made zero false positives.

This is the trade a general-purpose detector makes, and why a tuned, deterministic engine matters for clinical text: over-redaction isn't free — it destroys the meaning you were trying to preserve.

NHS numbers were tested in spaced, unspaced and dashed forms; dates of birth were introduced by varied keywords (Date of birth, DOB:, D.O.B., Born on).

Against Presidio, both engines reach 100% on NHS numbers, patient and relative names, dates, phone and email. They split on the two UK-specific identifiers Presidio has no recogniser for:

A redactor that flags everything is useless — it destroys the clinical record. Each note is seeded with false-positive bait. The engine correctly preserved all 1,329:

Accuracy is only half the story. Redacta is a 15 KB deterministic engine — no model to download, no server to call. That's what lets it run inside the iPhone app and the browser, so the text never leaves the device. A model-based detector has to load hundreds of megabytes and, in practice, runs server-side — which means sending the unredacted text away to redact it.

Redacta's bar is the hairline on the left — 15 KB is a rounding error next to a 560 MB model. Speed measured with Node on a laptop-class core; on-device in the app it runs in JavaScriptCore. Presidio figures are the en_core_web_lg model footprint, run as shipped.

It's fully reproducible — fixed seed, one command. The corpus is exported to corpus.json so the Presidio baseline scores on byte-identical input with the same rule (presidio_baseline.py).

These numbers describe the engine within its deterministic scope. They are high because the engine is regex- and checksum-based and near-exhaustive on the patterns it targets — the benchmark verifies that and probes the boundaries. The real limits:

Free-text names are out of scope. A name with no anchoring title, salutation, label or relationship word — "Patricia returned today…" — is not caught (0 of 50 in this run). This is by design for an on-device deterministic engine: the app prompts you to review, and the agent-skill layer adds AI reasoning for these cases.

It over-redacts in ambiguous cases — the safe direction. A 10-digit reference number formatted like a phone number, or "Sister [Name]" (a UK nurse title the engine reads as a relative), can be redacted even when it isn't personal data. Erring toward redaction is the safer error.

The corpus is Redacta's home turf. It's UK clinical text, the domain Redacta is tuned for, so a perfect score is expected — that's why the Presidio comparison matters more than the absolute number. Presidio is general-purpose and was run as shipped (default recognisers); a clinical-specific configuration would close some of the gap, though not the UK-identifier coverage.

Synthetic data is tidier than reality. Real letters carry OCR noise and unusual layouts, so real-world recall — especially for names — will be lower. Redacta is a strong first line of defence, not a guarantee: always review the output before sharing text.