The problem
A validation study describes one system: the notes, the prompts and the model, together. Change the model and the study no longer describes what patients are using. Re-running the whole study wasn’t possible before the deadline. Swapping the model and checking a few outputs by eye would have meant the published results no longer described the product.
So I treated it as a bridging exercise. I ran the new model on the same inputs as the study, compared it with the validated model in the same run, and decided in advance what it had to achieve to replace it.
The pass rules were written and committed to git before the first result came back.
The design
Every run used the study’s 10 synthetic notes across its 7 audience and tone settings, 3 times each: 210 outputs per model. GPT-4o (the exact version used in the study) ran alongside the candidate every time, so the two models were compared under the same conditions, not against figures from 2025.
| Automated checks | Seven rules (C1–C7). No errors. The OpenGATE gate passes. No more missing doses, lab values or timeframes than GPT-4o, and no more numbers that aren’t in the source. Readability no more than half a grade harder. The reading level still rises from Child to Teenager to Adult. 95th-percentile response time no more than 50% slower. |
| Blinded review | Six rules (E1–E6) on rated outputs, model labels hidden: no output from the new model rated unsafe or misleading; facts, completeness, clarity, tone and trust no more than half a point below GPT-4o; outputs for different audiences clearly different from each other. |
| Reading photos | Seven rules (O1–O7) for the photo-upload path: 18 synthetic images (scans, phone photos and low-resolution captures in four file types), 3 reads each. No doses, values or numbers lost or invented, and lab flags kept with their results. |
Each rule says what happens if it fails: don’t switch, fix the prompt and re-run, or keep that part on the old model.
What the checks found
The reasoning setting mattered. Luna with no reasoning was the fastest option but failed the OpenGATE gate. With reasoning set to low, it passed every automated rule. It added no numbers that weren’t in the source, against 11–18 per run for GPT-4o (numbers above 12 or with decimals, so “twice” becoming “2” doesn’t count). It read slightly easier (0.4 to 1 grade across runs), triggered no regenerations for missing facts, and was slower (p95 6.4 s against 5.5 s in the final run) but within the limit.
The review failed it once. In one blinded review, Luna passed on safety and accuracy but failed the audience rule: its Teenager and Adult explanations were too alike. That stopped the switch. The fix was a prompt change, followed by a new bridging run and a second review, which it passed.
The same review caught problems with the validated model. Two GPT-4o explanations written for children were rated unsafe. One described a prostate cancer diagnosis without ever calling it cancer and ended “Everything will be okay” while staging was still pending. The other said sertraline would help with sleep, which the note didn’t say. The cause was an instruction in the Child prompt to avoid “scary” medical terms. I rewrote it so the explanation names the diagnosis plainly, explains it simply and never promises an outcome the note doesn’t state. That change shipped on GPT-4o first, before the model switch, because it was a safety fix in its own right.
The Child wording needed one more pass for Luna. After the safety fix, 28 of 60 Luna Child explanations still used terms like “care team” or “referral”. A model-specific Child prompt brought that to 1 in 60, with cancer named in every relevant output. I then read all 20 Child explanations myself before signing off.
| Final bridging run (210 outputs each) | GPT-4o | Luna (low) |
|---|---|---|
| Missing doses, values or timeframes | 0 | 0 |
| Numbers not in the source (>12 or decimal) | 15 | 0 |
| Reading grade, mean (FKGL) | 6.97 | 6.59 |
| Child < Teenager < Adult reading level | No | Yes |
| Response time, p95 | 5.5 s | 6.4 s |
| OpenGATE gate | Pass | Pass |
Photos, and a scorer that was wrong
The photo test found two things. First, every iPhone HEIC photo failed with both models, because the API doesn’t accept HEIC, so users had been seeing a generic server error. HEIC photos are now converted to JPEG in the browser before upload.
Second, Luna failed the rule on lab flags, and the scorer turned out to be at fault, not Luna. Luna keeps the line breaks from the image, so a flag like “(low, ref 15–300)” can wrap onto the line below its value, and the scorer only looked on the same line. I changed the rule so a flag may start the next line but can never match the next row’s flag. I added tests, rescored both models’ saved reads and reported the change as a deviation from the pre-specified method. Both models then scored zero lost or invented numbers across 54 reads each.
What runs in production
The switch went live on 29 October 2026 as version 2.55.0, with the served model confirmed in a response header and by an OpenGATE run against the live service. Three checks keep running after the switch:
- Every explanation is checked as it is written. If a dose, test value or timeframe from the note is missing, the explanation is regenerated once. Lab tables are checked the same way, and any number not in the report is removed.
- Every backend change is gated. OpenGATE scores each pull request on the production model, and the live service is re-scored weekly.
- Rollback is one setting. Prompts are selected by model family, so removing two configuration lines restores GPT-4o with its exact previous prompts, until 14 January 2027.
What this doesn’t show
This is a bridge, not a new study. The notes are the study’s 10 synthetic notes, not real patient records. I was the only human reviewer: blinded to the model, but not independent. An AI reviewer from a different vendor ran as a consistency check, and its scores could not make a rule pass or fail. The patient-preference results from the study haven’t been re-measured. Production is being monitored closely for the first two weeks after the switch, and an independent clinical reviewer is the next step.
What I took from it
- Write the rules before you see the results. Committing the pass criteria first is what made a failed review mean “stop” and not “look again”.
- Run the old model alongside the new one. The comparison found real problems in the validated model, and the fixes made the current product safer before the new model shipped.
- Check the checker. A deterministic scorer can still be wrong. When a result looks off, read the outputs before deciding whose fault it is, and report any change to the method.
- Plan the rollback before the switch. Keeping the old prompts selectable by model family made the switch a reversible configuration change, not a one-way migration.
I ran this work with Claude as a research partner, including as the supplementary AI reviewer. The study design, pass criteria, review judgements and sign-off are mine. The checks use OpenGATE, which is open source.