AI Scribes Are Writing the Wrong Drug Into NHS Records
A patient’s test result said “null demyelination.” The AI scribe transcribing the consultation dropped the word “null.” The record now said the patient had nerve damage of the kind associated with multiple sclerosis. It said the opposite of what the result showed.
That case is one of several documented by Healthwatch England, the statutory patient watchdog, in a warning issued at the end of August about AI scribes in NHS settings. Others: a scribe that substituted a different drug with a similar name for the one actually prescribed; a consultant’s instruction to obtain a repeat migraine prescription that vanished from the notes entirely; a summary recording that a doctor had prescribed Prozac when the drug had been neither prescribed nor discussed.
Twenty-seven different AI scribing products are currently in use across the English NHS. The errors above were caught because patients happened to read their own records. Healthwatch’s observation about the ones that are not caught is the sentence worth sitting with: “These inaccuracies may persist in their records if the patient doesn’t catch them.”
Table of Contents
The Category Nobody Wanted to Define
The regulatory position is the part of this story that generalizes.
The Medicines and Healthcare products Regulatory Agency has not classified AI scribes as medical devices. Guidance published in August draws the line between passive transcription systems, which are not devices, and systems that suggest diagnoses or treatments, which may be. The distinction is coherent in principle. In practice it creates a gap that a product can sit inside comfortably: a tool that merely writes down what was said is not making a clinical judgment, so it falls outside the safety-testing regime that governs devices.
Except that a clinical record is not a passive artifact. It is the input to every subsequent decision about that patient. A transcription error that inverts a diagnosis does not stay a transcription error. It becomes the basis on which the next clinician prescribes, refers, or declines to investigate.
“This is the most consequential regulatory question in applied AI right now, and it is being decided quietly, product by product, in guidance documents almost nobody reads,” says AI expert and consultant Hassan Taher. “The question is whether you regulate a system by what it claims to do or by what depends on its output. Scribes are being regulated by the first standard. They should be regulated by the second. A tool that produces the authoritative record a clinician relies on is participating in the clinical decision whether or not it renders an opinion, and the fact that it never says the word ‘diagnosis’ does not change what happens downstream when it gets a drug name wrong.”
Where the Errors Come From Is Not Mysterious
The failure modes on display are the well-characterized failure modes of speech-to-text systems built on generative models, and none of them are surprising to anyone who has evaluated one.
Dropping “null” from “null demyelination” is a fluency artifact. Speech models are trained to produce plausible, well-formed language, and in clinical audio full of noise, accents, overlapping speech, and specialist vocabulary, the model resolves ambiguity toward what sounds most like normal speech. Negations, qualifiers, and hedges are precisely the words most likely to be smoothed away, because they are short, unstressed, and frequently absent from the surrounding statistical pattern. They are also the words that carry the clinical meaning.
Drug-name substitution is the same mechanism aimed at a different target. Pharmaceutical names are phonetically dense and deliberately similar within therapeutic classes. A model choosing the more common of two similar-sounding drugs is behaving exactly as designed and producing a dangerous output.
And the Prozac case — a medication appearing in a summary that was never mentioned in the room — is generation rather than transcription. A summarization layer producing a clinically coherent note filled in a plausible detail. This is the behavior that makes these tools useful and the behavior that makes them unsafe, and it is not separable into two settings. It is the same failure Hassan Taher has described in the challenge of hallucination in AI chatbots, arriving in a setting where the output is a medical record.
The pattern connects to something Taher has documented elsewhere: systems that clear extraordinary benchmarks while failing at tasks as ordinary as reading a clock. Capability in these systems is jagged. Performance on an aggregate accuracy metric says very little about behavior on the specific inputs where the cost of being wrong is highest.
The Consent Problem Is Separate and Also Unresolved
Healthwatch’s warning followed survey work conducted with YouGov in April, polling 4,039 adults alongside 44 qualitative accounts from patients. The findings on consent are stark.
Nearly 90% of respondents who had recent appointments were not aware an AI scribe was being used. Eighty-one percent said they wanted to be told and asked before one was. Forty-eight percent were comfortable with a scribe present during a routine health check — but that fell to 29% for sexual health, 28% for mental health, and 23% for consultations about domestic abuse. Sixty-nine percent said they would feel more comfortable if the clinician committed to checking the record for accuracy.
Public opinion on rollout is close to evenly split: 38% supportive, 37% opposed, with nearly twice as many strongly opposed (21%) as strongly supportive (11%). Healthwatch recommended prominent information at the point of booking, an honored right to have the scribe switched off for sensitive consultations, and consistent national standards from regulators.
The gap between 90% unaware and 81% wanting to be asked is the entire governance failure in two numbers. The technology was deployed at scale into consultations before the population using those consultations was told it existed.
What Transfers to Other Industries
Very few organizations run clinical services. Nearly all of them are deploying something structurally identical to an AI scribe: a system that observes a human process, produces a record of it, and hands that record to someone who will act on it. Meeting summarizers feeding project decisions. Call transcription feeding compliance files. Document extraction feeding underwriting, claims, or contract review.
Three lessons transfer directly.
“It only transcribes” is not a risk assessment. The question is not what the system decides. It is what decisions depend on its output. Any system whose output becomes the record of record carries decision-grade risk regardless of how passive its function is described as being.
Aggregate accuracy is the wrong metric. A scribe at 98% word accuracy sounds excellent and can still invert a diagnosis, because the 2% is not randomly distributed. Evaluation has to be weighted toward the categories where errors are consequential: negations, quantities, names, dates, and dosages. Test the failure modes, not the average.
Verification has to be assigned to someone with time to do it. The implicit safety model for AI scribes is that a clinician reviews the note. The actual justification for buying one is that clinicians do not have time to write notes. Those two propositions are in direct tension, and it resolves in favor of whichever one the schedule enforces. That unresolved tension is a recurring reason deployments underdeliver, a pattern Hassan Taher has examined in the MIT finding that most AI projects fail. In the NHS cases, the reviewer who caught the error was the patient — a control that exists by accident and covers only the patients who read their records.
“The pattern I keep seeing is that the accountability for AI output gets assigned to the person with the least capacity to exercise it,” Taher says. “The productivity case for the tool is that a busy professional no longer has to do the task. The safety case for the tool is that the same busy professional carefully checks it. You cannot hold both. Organizations that want the productivity have to fund the verification separately — a second reviewer, an automated cross-check, a structured audit sample — or accept honestly that errors will reach the record. What does not work is writing ‘human in the loop’ into the deployment plan and treating the question as settled.”
This article discusses healthcare technology and regulatory developments for informational purposes. It is general commentary and not medical or legal advice.
