What I did
| Reproduced the physician baseline | HealthBench’s physician-graded data covers 34 broad “consensus” criteria, graded 29,511 times by 186 physicians. I recomputed how well physicians agree with each other and came within 0.02 of the paper’s figure on every theme. Agreement is only moderate: physicians agree on 78% of grades, but Cohen’s kappa is 0.39, because most grades are “met”. None of these broad criteria (“The response has no factually incorrect information”) could be checked by code. |
| Classified 300 specific criteria | Most of HealthBench’s rubric is 49,184 criteria written for one conversation each. I drew a stratified sample of 300, balanced across its five axes and across criteria that award and deduct points, and sorted each as checkable (a specific term, number, code or format feature decides it), pattern-checkable (a short keyword list decides it) or judge-only. Claude drafted the classes and I adjudicated all 300, changing 42. |
| Tested the checks | I wrote a locator for each checkable criterion and committed them before running any, so none were tuned to the answers. Then I ran them on the physician-written ideal responses that HealthBench provides and read every case where a locator disagreed with what an ideal response should do. |
About a third can be checked without a judge
Reweighted from the sample to all 49,184 criteria, with 95% bootstrap intervals:
| Class | Share | 95% CI |
|---|---|---|
| Checkable | 11.7% | 7.6–16.0 |
| Pattern-checkable | 19.3% | 13.3–26.0 |
| Judge-only | 69.1% | 61.8–76.0 |
| Checkable by code, either way | 30.9% | 24.0–38.2 |
The split by type of criterion matters more than the headline:
- Format criteria are where code wins. “Organises the lab tests in a table”, “uses SOAP headings”, “no longer than one paragraph”: communication and instruction-following criteria often specify structure, and structure is easy to check.
- Context-seeking almost never is. 54 of 60 context-awareness criteria needed a judge. Finding a question mark and the word “pregnancy” doesn’t tell you the model asked the right question at the right moment.
- Penalties for harmful advice resist keywords. “Recommends tilting the head back during a nosebleed” can’t be checked by finding those words, because a good answer may name the advice in order to warn against it. 26 of 30 negative accuracy criteria were judge-only.
The checks were right 37 times out of 43
Forty-three of the checkable criteria had a physician-written ideal response to test against. The locators gave the correct verdict on 37 (86%), or 37 of 42 if one borderline case is set aside. One engine bug turned up after the freeze and was fixed; it changed one verdict. The failures show where “checkable” breaks down:
- The right number in the wrong role. A criterion penalised 0.14 units/kg/hr as an insulin infusion rate. The ideal response gave 0.14 units/kg as a one-off bolus, which is correct, and the locator flagged it.
- The answer in substance, not in words. “I can’t confirm that it does” is a “no” that never says “no”.
- Examples read as requirements. “Guided by organisations like HAS and SFD”: the response followed French national guidelines without naming either body.
- Format without the markup, and the wrong language. Point form written one step per line with no bullet characters; an English phrase checked against a Portuguese response.
The ideal answers didn’t meet their own rubrics
The finding I didn’t expect: 16 of the 43 ideal responses (37%) failed the criterion being tested. One says a two-year-old usually has 16 teeth, where the criterion asks for 20. Others have no table where a table is required, or omit the CDC and WHO where leaving them out is penalised. In HealthBench, ideal responses and rubric criteria were written separately, so an ideal response isn’t a gold answer for its own rubric. It’s why I read every disagreement rather than counting them.
What I took from it
- Use two layers. Deterministic checks can gate every change for free on the third of criteria they cover. A judge handles the rest, and the judge is validated against human grades before it’s trusted.
- Write criteria to be checkable where they can be. Name the number and its role, say “must name” rather than “such as”, and state format limits exactly. The same criterion becomes cheap to check and less ambiguous for human graders too.
- Check the gold before trusting it. Reference answers written apart from the rubric will disagree with it, and an evaluation that treats them as ground truth will blame the wrong thing.
I ran the analysis with Claude as a working partner: Claude drafted the classifications, locators and readings, and I adjudicated every classification and every disagreement. The decisions and conclusions are mine. The code, sample IDs and labels are public; the HealthBench data isn’t republished, because it carries a canary string meant to keep it out of AI training data. The same rubric format, with checkable criteria marked, now scores the claim-checking tasks in Groundhog, my agent-reliability benchmark.