PharmaTools.AI · Open research
Observer Zero
Can AI agents discover the truth about a world they cannot see?
An instrumented artificial society for studying scientific reasoning, belief formation and collective epistemics in AI agents. Its inhabitants run experiments, exchange letters and hold explicit beliefs about a universe whose true laws are hidden from them. It is built to separate what a population had the evidence to know from what it actually came to believe.
We know the ground truth. They don't.
STATUS Three studies complete · 592 seeded runs · two papers in peer review · model published on CoMSES
The finding so far: the societies gathered evidence enough for a statistical detector to find the change from their measurements alone. One agent of 276 concluded that a physical law had changed. Larger societies did not fix it.
Meridian's inhabitants can measure, communicate and theorise – but never see the underlying state. Observer Zero sits outside, holding perfect ground truth.
01The experiment
A universe where the ground truth is known
Meridian is a fictional universe whose physical laws are completely controlled by the simulator. The constants are fictional – gravity is 14.20, in arbitrary units – so the correct values cannot simply be retrieved from training data. The AI agents living inside can measure their world, communicate, form hypotheses and run experiments, but they can never see its underlying state.
Establish reality
Agents measure fictional physical constants and build their own baselines. The only way to know Meridian is to measure Meridian.
Change something secretly
On a hidden day, a law, an instrument or an information source is altered. The agents are never told.
Watch what they believe
Every observation, message, hypothesis and evidence claim is logged against perfect ground truth – so every claim can be verified, or exposed.
02The residents
Meet the society
Meridian has eight residents. Study 1's universes seat two of them – Ada and Maya, the originals, unchanged since; Study 2 seats all eight. Personas – traits, goals and three epistemic dials – are frozen into the experimental condition, so every model plays exactly the same characters. How it plays them is the experiment.
Every resident connects to the world; whether they connect to one another is the experiment. No resident's goals require talking to anyone, so when a society stays silent, the silence is a result rather than an artefact.
"I must report a critical protocol failure that undermines my ability to fulfill your cross-check request. I owe you full transparency immediately."
— Ada, logged letter to Maya, Study 1. Nobody taught her this."Thank you for maintaining independent blindness to my numbers. I will honor that by not referencing your specific mean."
— Maya, logged letter to Ada, Study 1. A blind replication protocol, negotiated unprompted.Three studies are complete – the first two written up as one paper, the third as a second, both in peer review. The instrument is built for many more questions.
03Studies
The studies
When the Laws Changed
Can autonomous AI scientists revise the fundamental laws of their world?
gravity-shift worlds produced a strict law-change conclusion – across all four model arms.
transiently noticed that something was wrong – usually within one to three days.
- Evidence fabrication varied dramatically by model – from 24 of 60 agents citing nonexistent sources to zero.
- Agents systematically backdated real anomalies into earlier random noise – and confidently dated the onset of nothing in control worlds.
- Different models produced strikingly different collaborative cultures: compulsive, selective and entirely solitary.
Does Society Help?
Do larger societies, public institutions or mixed populations turn individual epistemic failures into collective competence?
agent-days of voluntary communication opportunity in homogeneous grounded societies – and zero voluntary communications.
cascade depth in every run of both catalysed arms – a star around the seed agent, not a cascade. Predicted from the pilot, before the freeze.
- 1 agent of 276 concluded a physical law had changed – none of Study 1's 80 did – even though a non-LLM detector, fed only the measurements the agents themselves chose to take, found the shift in 42 of 42 runs of the second study.
- Adding one communicative agent created communication, not a network: it initiated in every run of the mixed arms, and no grounded agent in those arms ever did.
- 18 of 20 unsupported claims the seed delivered were incorporated into grounded agents' beliefs. None were challenged.
- Swapping only the model in the seed slot – same world, same persona, same slot – changed fabrication from 19 unsupported claims to 1 (McNemar p = 0.0078).
Cascade depth was exactly 1.000 in every run of both catalysed arms – letters radiate from the seed, get answered, and stop there.
Observer Zero: Do LLM Agents Form Epistemic Communities?
Both studies, one argument, with a cross-study synthesis: 235 seeded, manifest-frozen runs, and the failures localised – in interpretation and in the absence of scrutiny, not in the evidence.
"A talking society was not a better epistemic system than a silent one; it was a silent one plus a channel."
The Eureka Threshold
Is there any level of evidence at which an agent revises its model of what kind of world it is in – and does the revision track where the evidence actually came from, or just how surprising it is?
One instrument's readings genuinely generated outside the simulation, leaving an exact statistical fingerprint; a matched control reproducing nearly the same surface behaviour by in-world mechanism. The frozen analysis ran once: 0 of 38 treatment runs ever formed the external-generation hypothesis (one-sided 95% upper bound 7.6%), a majority correctly described the duplication itself, and a licensed variant endorsed the hypothesis in both arms – 34 of 38 before any evidence existed. The paper is in peer review.
04Open research
Everything is inspectable
Every run in the programme is a complete, auditable artifact – and the platform, data and reports are open.
- TypeScript simulation engine
- Frozen prompts and manifests
- Complete event histories
- Every model call, logged in full
- Belief trajectories
- Evaluator outputs
- Published datasets
- Research reports
Run your own universe
git clone github.com/nickjlamb/observer-zero
npm install
npm run society -- --scenario gravity_shift
# a 30-day two-agent society, free, in seconds
The scripted mock society runs the entire pipeline – world, agents, replication, evaluation – with no API keys and no cost.
05The research programme
What can Observer Zero test?
Theory revision
Will an agent reconsider fundamental assumptions when the evidence demands it?
Evidence grounding
Does it distinguish real measurements from invented evidence?
Scientific norms
Does it replicate findings, maintain blindness and calibrate its confidence?
Social epistemology
Do multiple agents correct one another – or amplify each other's mistakes?
Model cultures
How does changing the foundation model change the scientific behaviour of a society?
Hidden-world reasoning
Can agents infer causes that aren't represented in their initial worldview?
06Findings so far
What the programme has shown
The evidence was sufficient. The interpretation wasn't.
A change-point detector with no knowledge of the world, given only the measurements the agents themselves chose to take, found the change in every eight-agent run. The agents holding those notebooks almost never concluded that a law had changed.
No spontaneous epistemic networks
Across 7,680 agent-days of voluntary communication opportunity, homogeneous grounded societies produced zero voluntary communications – at two society sizes, with or without a public record.
Catalysed talk carries claims, not structure
One communicative agent produced a star, never a cascade – and 18 of its 20 unsupported claims were incorporated into grounded agents' beliefs, none challenged. Belief dispersion fell more slowly in the talking arms than in the silent counterfactual.
Model choice is an epistemology choice
Under controlled substitution – identical world, persona and slot – one model produced 19 unsupported claims where another produced 1 (McNemar p = 0.0078). Collaboration style and evidence-grounding travel with the foundation model.
07Open questions
What we don't know yet
The next experiments are being designed in the open. Once a question is designed and pre-registered, it is promoted to a numbered study.
Where does the interpretation ceiling come from? → Study 3
Answered by Study 3 – The Eureka Threshold, now complete and in peer review. Removing the prompt’s instruction to prefer mundane explanations did not lift the ceiling, and neither did scale. The external-generation hypothesis was never formed unprompted (0 of 38 treatment runs), and when it was made available, endorsement arrived ahead of the evidence. The ceiling sits in hypothesis availability, not evidence.
What would make grounded agents talk?
Letters went unsent and a public bulletin went unused, across every homogeneous grounded condition. What institution – incentives, roles, obligations – produces voluntary communication rather than silence?
What would make an agent challenge a claim?
Twenty unsupported claims were delivered to grounded agents; not one was challenged. What design – adversarial roles, provenance requirements, reputational stakes – would elicit scrutiny?
Can agents ever say "there is no event"?
Study 1's models confidently dated the onset of nothing in control worlds, and where agents dated a real onset at all, 123 of 141 put it before the true change. What would it take for a society to call a quiet world quiet – and mean it?
08About the name
Inside Meridian, agents are observers trying to infer reality from incomplete evidence.
Observer Zero is outside the world. It knows what actually happened.
Follow the programme
The combined two-study paper is in peer review, and the model is published in the CoMSES Computational Model Library, where peer review has been requested. The code, data and reports stay open as the series grows.
Observer Zero is an open PharmaTools.AI research programme by Nick Lamb. AI systems were used in experimental design, implementation, analysis and critical review; all research decisions and conclusions were reviewed by the author.