In the spring of 2023, a medical student walked into my office convinced that the chatbot everyone was suddenly talking about was either going to replace us or get someone killed. I told her we didn't actually know which, and that the fix for not knowing was to run the experiment.
That experiment became one of the first peer-reviewed clinical evaluations of a large language model: Accuracy and Reliability of Chatbot Responses to Physician Questions, which we published in JAMA Network Open in October 2023 (Goodman et al.), a cross-sectional study I co-authored. The questions came from thirty-three practicing physicians across seventeen specialties, and the grading came from those same physicians. It is now three years later, the models are unrecognizably better, and I think it is worth saying plainly what that study got right, what it got wrong, and what I would measure if I were starting it today.
What we actually did, and why
The design choice that mattered most was mundane: we used real questions from real physicians, not a board-exam item bank. Benchmark questions have clean answers. Clinical questions are messy, context-dependent, and frequently have no single right answer at all. We asked attending physicians to submit questions they had genuinely encountered, fed them to the model, and graded the responses for accuracy on a six-point scale and completeness on a three-point scale.
The headline result was not "the model is good" or "the model is bad." It was that the median response was reasonably accurate and reasonably complete, and that the variance was the problem. The same system that produced a genuinely useful answer to one question produced a confidently wrong one to the next, with no change in tone, no hedging, no signal to the reader that the ground had shifted under them. We flagged that a small number of responses, if acted on, could have caused harm.
That same year I co-wrote a short companion piece in Med, On the cusp (Goodman, Patrinely, Osterman et al.), arguing that the technology was arriving in healthcare whether or not the field was ready, and that the responsible move was to evaluate it in the open rather than ban it and watch it get used in the shadows.
What held up
Two things from 2023 have aged well, and they are the two I would underline for anyone deploying an LLM in a clinic today.
The first is that fluency is not accuracy, and the gap between them is invisible to the reader. This was the central finding, and it has only become more important as the models have gotten better. A weak model that writes badly gets double-checked. A strong model that writes beautifully gets trusted, and the rare wrong answer rides in on the same polished prose as the right ones. The better the writing, the higher the burden of verification, not the lower. Three years of capability gains have not repealed that.
The second is that you cannot evaluate a clinical model without a clinical ground truth to check it against. Our whole method depended on physicians who knew the right answer grading responses where the right answer was knowable. The moment you move into questions where the evidence is genuinely unsettled, which is most of oncology most of the time, "accuracy" stops being a clean number and starts being a judgment call. That is not a flaw in the chatbot. It is the actual shape of the problem, and any evaluation that pretends otherwise is measuring the wrong thing.
What changed
Plenty. The 2023 study evaluated a single general-purpose model answering from its training data, with no access to the patient's chart, no retrieval over current guidelines, and no tools. Almost none of those constraints hold in 2026.
- The models got dramatically better at the easy-to-grade questions, the ones with a defensible single answer. That is real progress, and it is also the least interesting part, because those were never the questions that worried me.
- Retrieval changed the failure mode. Grounding a model in current guidelines and the actual record moved the dominant error from "confidently wrong from memory" to "confidently wrong about a retrieved source it misread." The errors got rarer and harder to catch, which is not the same as safer.
- The deployment surface moved. In 2023 the clinical question was "should a physician paste this into a chatbot." By 2026 the model is inside the ambient scribe, the in-basket drafting tool, the order-entry suggestion. Nobody is consciously consulting it. That is a different risk profile, and our 2023 design, physician asks, physician grades, does not capture it.
So the most-cited number from our paper, the accuracy score, is the part that dated fastest. The method dated slowest. If you cite the 2023 accuracy figure as if it describes a 2026 model, you are quoting a stopped clock. If you adopt the 2023 method, real questions, clinical graders, explicit attention to variance and worst-case harm, you are doing the only kind of evaluation that still tells you anything.
The policy debate it seeded
The part I did not anticipate was how the paper would get used. It landed at a moment when health-system leaders, specialty societies, and regulators were all reaching for evidence, and a peer-reviewed clinical evaluation was scarce. The study got cited far more in governance documents and policy arguments than in follow-on technical work, and frequently to support whatever the citing party already believed. Enthusiasts quoted the median accuracy. Skeptics quoted the harmful outliers. Both numbers were in the same paper, which was rather the point.
The durable lesson there is about evaluation cadence, not models. A clinical LLM is not a device you validate once and certify. It is a moving target: the underlying model is updated without notice, the retrieval corpus changes, the clinical context drifts. An evaluation is a photograph of a system that no longer exists by the time the paper prints. Governance that treats a single published accuracy number as a safety guarantee is governance built on a photograph. The institutions getting this right are standing up continuous local evaluation, measuring the model they are actually running, on their own patient population, on an ongoing basis, rather than importing someone else's 2023 snapshot.
What I would measure today
If I were designing the study again in 2026, three things would change.
I would stop scoring the model in isolation and start scoring the physician-plus-model system, because that is what actually touches patients now. The right question is no longer "how accurate is the chatbot" but "does the clinician catch it when the chatbot is wrong", and the unsettling early signal is that better writing makes us catch less. I would measure failure modes before central tendency: not the average, but the shape and cost of the tail, because in medicine the tail is where the harm lives. And I would build the evaluation to run continuously against the locally deployed model rather than publish once and move on, because a one-time number is obsolete the day the vendor pushes an update.
None of this requires a new model. It requires the boring infrastructure to measure the one you have, which, not coincidentally, is the same argument I make about AI in oncology generally and about the data layer underneath it. The 2023 study's real contribution was never the score. It was demonstrating that you can hold a clinical LLM to an honest, physician-graded standard at all. Three years on, the models have outgrown our numbers. The standard is what's worth keeping.