The Note Is Yours, Even When AI Drafts It
An AI scribe can hand you back the evening you used to spend charting. That's a real gain, and you don't have to feel guilty about wanting it. Being more present in the room and finishing your day on time are good outcomes for you and for the people you treat.
But the moment an AI draft becomes part of a client's record, the question changes from "did this save me time?" to "who is accountable for what this note says?" That record might one day be read by the client, a licensing board, an auditor, or a court. You need to be able to stand behind every line of it.
A black-box scribe that nobody reads is a liability. A smarter model or a lower error rate will not fix that. The safeguard that holds is a licensed human who reads the draft against their own memory of the session and signs it. That principle is the fourth plank of our No AI Therapist Pledge (the clinician is always the author of record), and we think it matters more than almost anything else an AI EHR can promise.
What a "Black-Box Scribe" Actually Means
The failure mode is specific. An AI scribe listens to (or transcribes) a session, generates a summary or a note, and that text drops into the chart. In a black-box setup, no human ever compares the generated text to what was actually said. The draft becomes the record by default.
That would be manageable if AI transcription only ever misheard: a wrong word here, a garbled name there, the kind of error you'd catch on a quick read. The documented risk is stranger and more serious. Sometimes AI transcription invents words, sentences, and events that were never spoken at all.
A boundary: this post is about AI scribes, tools that document a session. AI "therapist" chatbots that try to deliver care are a different technology with a different set of harms, and conflating the two muddies the argument. The evidence below is specifically about speech-to-text and note generation.
The Evidence: AI Transcription Invents Words No One Said
The clearest study on this comes from researchers at Cornell, the University of Washington, NYU, and the University of Virginia. Titled "Careless Whisper: Speech-to-Text Hallucination Harms," it was presented at the ACM Conference on Fairness, Accountability, and Transparency (FAccT) in June 2024. The team examined OpenAI's Whisper, one of the most widely used open speech-to-text models.
The Cornell release reports that roughly 1% of transcriptions contained entirely hallucinated phrases: passages of text the model generated that no one had said. In one example, the tool correctly transcribed a single sentence and then invented several more sentences of violent content, with words like "terror," "knife," and "killed" that were nowhere in the original audio. (Coverage of the same study by the Associated Press put a count on it: 187 hallucinations across more than 13,000 clear audio snippets.) Hallucinations clustered around pauses and silences, which means they disproportionately affected speakers with aphasia or other speech differences, exactly the kind of clients a behavioral-health practice often serves.
The pattern shows up beyond the lab, too. An Associated Press investigation by Garance Burke and Hilke Schellmann, published October 26, 2024, gathered accounts from people working with Whisper in the field. A University of Michigan researcher found hallucinations in 8 of every 10 public-meeting transcriptions he reviewed. A machine-learning engineer found them in about half of more than 100 hours of audio. A developer found them in nearly all of 26,000 transcripts he created. These are different corpora measured different ways, so they do not add up to a single clinical hallucination rate. Together they show a recurring, real-world behavior.
The healthcare example is the one every clinician should sit with. As the AP reported, a Whisper-based medical scribe from the company Nabla has been used by more than 30,000 clinicians and 40 health systems across roughly 7 million medical visits. And that tool deletes the original audio recording, for stated data-safety reasons, which means there is no ground truth left to check the transcript against. A former OpenAI engineer summed it up: "You can't catch errors if you take away the ground truth." OpenAI itself has warned that Whisper should not be used in "high-risk domains." A clinical record is the definition of a high-risk domain, and a tool that destroys the only thing you could check the note against is the definition of a black box.
A note that no human read, generated by a tool that deleted the audio, is an unverifiable claim about what happened in the room.
On scope: the Nabla and Whisper details above are an industry example of the black-box pattern, drawn from public reporting. They describe how some tools in the market behave. Coral's own pipeline is described separately below.
Why "The Model Will Get Better" Isn't the Answer
The natural reply is that these models improve every year and the error rate keeps dropping. That is true, and it misses the point.
A lower hallucination rate is not a zero hallucination rate. And in a clinical record, the cost of the rare invented line is asymmetric. A medication that was never prescribed. A symptom the client never reported. A risk statement whose meaning flips from "denied suicidal ideation" to something else entirely. One of those errors, sitting unread in a chart, can do more damage than a hundred saved hours can repay.
You also cannot quality-control your way out of this if no human is in the loop and the source audio is gone. There is nothing to QA against. The only reliable check on a draft is a person who was actually in the room, reading the text against their own memory of the session. No model improvement removes the need for that person; it only changes how often that person catches something.
The Signature Is the Safeguard
A clinician's signature on a note is more than a formality. It's the moment a block of text becomes a clinical and legal attestation. When you sign, you are saying: this reflects my professional judgment of what happened in this session. That sentence cannot be true unless a professional actually read it.
What the Law Already Assumes
The legal framing lines up with the clinical one. In July 2025, the Texas Medical Liability Trust published risk-management guidance on AI medical scribes (author Wayne Wenske), drawing on positions from the Federation of State Medical Boards. The core point: physicians remain fully responsible for the content of all medical documentation, regardless of how it was generated. Clinicians must review, edit, and sign AI scribe output before it enters the record. Automatic signatures on AI-generated content are discouraged. The responsibility for the record stays with the clinician, no matter how much of the draft the software produced.
One caveat: that guidance is physician-facing, and it speaks to medical boards rather than to a therapist-specific statute. Treat it as a principle rather than a regulation written for your license. The principle generalizes cleanly to every licensed clinician: the human professional owns the record.
It generalizes for a simple reason. AI can't hold a license. It can't be sued. It can't testify. When something in a record is challenged, accountability has to land somewhere, and the only place it can land is the human who signed. Sign-off is where professional responsibility lives.
How This Shows Up in Good Design
A well-designed AI EHR treats sign-off as the spine of the workflow. Our pledge states the commitment directly:
Our AI drafts, transcribes, summarizes, and suggests. It does not diagnose, it does not decide, and nothing it produces enters the record until a licensed clinician has read it and signed it. Every suggestion is reviewable and traceable.
Two things follow from taking that seriously, and we think any trustworthy AI EHR should be able to claim both.
The first is that a good system makes the draft easy to review against the truth: traceable, editable, reviewable. Deleting the audio does the opposite. The black-box scribe fails because it removes your ability to check, and a care-first design preserves it.
The second is that your data trains nothing. The draft works for the clinician whose data it is, within that clinician's boundary; the session is not pooled into a shared training set. Our pledge's second plank says the AI works on that clinician's behalf, within that clinician's boundary, and we will not feed your patients' data to a model that outlives the relationship it came from. In its own words, "today, the simple answer is: we don't." That is a present-tense commitment, scoped to today, and we describe it that way on purpose.
Here is how that works in CoralEHR, specifically:
- No stored recordings. The AI scribe stores no session recording, so there is no session audio to retain. And because every draft, however it is produced, is reviewed and signed by a licensed clinician before it enters the record, you remain the check on anything the AI generates.
- First-party model, no training on your data. The AI runs on Anthropic's first-party Claude API (never AWS Bedrock or a third-party aggregator) through a single audited code path under a Business Associate Agreement; your patients' data is not used to train models, and access is scoped to the clinician-patient relationship.
- Preliminary until you sign. An AI note is saved as a preliminary draft and does not become part of the clinical record until you sign it in a separate, deliberate signing step. There is no auto-sign.
These describe how the product is built today. (Separately, our pledge notes that we are pursuing independent third-party security attestation appropriate to healthcare; we don't claim those attestations as already held.)
What to Ask Any AI Scribe Vendor
You don't need to take anyone's word for this, including ours. Turn the argument into a short due-diligence checklist and put it to every vendor you consider. Get the answers in writing, in a Business Associate Agreement or your subscription terms. Marketing pages don't count.
- Does a licensed clinician review and sign before anything enters the record? If the tool auto-signs notes, walk away.
- Can I see and edit the draft against the source? Is the original audio or transcript retained so you can verify what the AI produced? A tool that deletes the source has removed your ability to catch its errors.
- Is my session data used to train any shared or foundation model? Get the answer in writing, and look for the difference between "we won't use identifiable data" and "we won't use your data at all."
- Does the vendor sign a BAA? This is the HIPAA floor. (Our deeper guide to vendor due diligence lives in the HIPAA guide for private-practice therapists, and you can pressure-test your whole stack with the free HIPAA Compliance Checklist.)
- Who is the author of record? The only acceptable answer is: you.
Care You Can Defend
Strip everything else away and the argument is short. AI can carry the paperwork. It cannot carry the accountability.
No vendor can promise that the model will never hallucinate. The safeguard that holds is a licensed human who reads the draft and signs it, working inside a system designed to make that review meaningful: the source stays checkable, the edits stay traceable, and the data trains nothing. That combination is what lets you get your evenings back and keep a record you can stand behind to a client, a board, or a court.
That's the trade we think is worth insisting on: let the AI do the drafting, and keep the signature, with the responsibility it represents, in the clinician's hands.
See how this works in practice:
- Read our No AI Therapist Pledge: including plank four: the clinician is always the author of record.
- Telling clients you use AI for notes: a care-first guide to disclosure and consent.
- HIPAA Compliance Checklist: a 35-item self-assessment to pressure-test your whole vendor stack.
- Explore CoralEHR: a HIPAA-compliant EHR built for private-pay behavioral-health practice.
Additional Resources
Internal:
- The No AI Therapist Pledge: our published commitments on AI and your data
- HIPAA Made Simple: A Private Practice Therapist's Guide: BAAs and vendor framing
- How to Tell Clients You Use AI for Notes: disclosure and recording consent
- HIPAA Compliance Checklist: interactive vendor and safeguards self-assessment
External:
- Cornell: AI speech-to-text can hallucinate violent language: the FAccT 2024 "Careless Whisper" study
- Associated Press (via TechCrunch): Whisper transcription tool has hallucination issues: the October 2024 investigation, including the Nabla deployment
- Texas Medical Liability Trust: Using AI medical scribes, risk management considerations: July 2025 guidance drawing on the FSMB
Frequently Asked Questions
Do I have to review AI scribe notes before they go in the chart?
Yes. Review is where the legal and clinical responsibility lives. Guidance the Texas Medical Liability Trust published in July 2025, drawing on Federation of State Medical Boards positions, states that the clinician remains fully responsible for the content of all medical documentation regardless of how it was generated. That guidance is physician-facing, but the underlying principle, that the licensed professional is accountable for the record, generalizes to all licensed clinicians. Review the draft against your own memory of the session, edit anything that's off, and sign only what you can stand behind. Automatic signatures are specifically discouraged.
Who is legally responsible for an AI-generated clinical note?
You are: the licensed clinician who signs it. AI can't hold a license, can't be sued, and can't testify. Per the FSMB-based guidance summarized by the Texas Medical Liability Trust (July 2025), the clinician remains fully responsible for the content of all medical documentation regardless of how it was generated; that responsibility doesn't transfer to the software just because a tool drafted the text. When you sign, you are attesting that the note reflects your professional judgment of what happened. That attestation is the whole point of the signature, and it can only land on a person.
Can AI transcription tools really make up things that were never said?
Yes, and it's documented. A study led by researchers at Cornell, the University of Washington, NYU and the University of Virginia, presented at the ACM FAccT conference in June 2024, ran more than 13,000 clear audio clips through OpenAI's Whisper. The Cornell release reports that roughly 1% of transcriptions contained entirely hallucinated phrases the model invented but no speaker said, and in one case the tool fabricated violent content that wasn't in the audio; Associated Press coverage of the same study counted 187 hallucinations across those clips. Hallucinations clustered around pauses and silences, which made them worse for people with speech differences. This is a documented failure mode of AI speech-to-text, which is why a human who was in the room needs to check the draft.
What makes an AI scribe a 'black box,' and why is that a problem?
A black-box scribe is one where the AI summarizes or transcribes the session, the text drops into the chart, and no human ever compares it to what was actually said. The most extreme version deletes the original audio, so there's no ground truth left to check against. As reported by the Associated Press in October 2024, one widely used Whisper-based medical scribe deletes the source recording 'for data safety reasons.' A former OpenAI engineer put the problem plainly: 'You can't catch errors if you take away the ground truth.' The fix is a human in the loop and a record you can verify.
What should I ask before choosing an AI scribe for my practice?
Ask five things, and get the answers in writing. (1) Does a licensed clinician review and sign before anything enters the record? (2) Can I see and edit the draft against the source, and is the original audio or transcript retained so I can verify it? (3) Is my session data used to train any shared or foundation model? (4) Does the vendor sign a Business Associate Agreement? (5) Who is the author of record, and is the answer 'me'? If a vendor auto-signs notes, won't sign a BAA, or can't tell you who the author of record is, walk away.
Arhaan Gupta
Co-founder and CEO, CoralEHR
I co-founded CoralEHR, an EHR for private-pay therapy practices, and I set up every new practice with them myself.
LinkedIn