- cross-posted to:
- hackernews@lemmy.bestiver.se
- cross-posted to:
- hackernews@lemmy.bestiver.se
At a recent appointment, Emily’s physical therapist (who knows some about her research) said, “Before we get started, there’s something I want to ask you about.” The something was an automatic “scribing” system their office is trialling for two weeks and deciding whether to purchase. These systems take in a (presumably audio-only) recording of the patient encounter and then output a draft patient note for the chart.
So what’s the big deal with “AI” charting? Here are nine reasons why we recommend refusing to consent to the use of scribing tools in healthcare settings:
I have never seen anything that starts with typing?
have also noticed it. I have on my to do list to look into these. hmu if you find anything. fuck knows what the training data is for. We have campaigns to divest HOOP from israel and I think AI should be included.
Smaller emrs for small behavioral health practices tend to have text only LLM - headway has scribe, therapynotes has therapyfuel, simple practice has one too and I forget the name. All of these work by taking written session notes and generating a proper progress note, e.g. “client reports more positive mood, anxiety remains triggered by x, practiced breathing exercises” gets extrapolated out into “the client presented with improvements noted in mood and session focused on mindfulness strategies to manage anxiety related to x” (more in depth than that but you get it)
In my evaluation the LLMs worked fairly well but not always, they tended to shift “writing the note” to “evaluate the note”. I don’t use it for my practice for several reasons, primarily the allure of skipping the evaluation step and ending up with notes that potentially have irrelevant hallucinations/outright falsehoods that get included because a clinician got lazy. As the article states I also believe the process of writing the note serves for reflection on the case and a time to consider conceptualization, eliminating that is certainly convenient but a potential harm to successful outcomes
When it comes down to it I simply don’t trust this in a server sided application. The only way I would consider it is if I could run a local model, generate the note on a pc that has no internet access, and then upload it. It’s granted that bc it’s entered into the EMR it could certainly be harvested for llm training because AI fucks are without ethics and above the law, but the chances of that are far lower. A workaround for this that solves both of my issues actually came from the psychotherapy subreddit back when I still used reddit: just use a local speech transcription (though confirm it happens locally, which is the case on apple devices post ios 16 or so)
WRT Israeli companies twofold scribe is currently the one that bothers me the most about signing up. They are headquartered in NY but a subsidiary of Ravel Technologies, an Israeli company. If you look up the developer info on the Google play store the given address for the twofold app is “4 Gordon REHOVOT, 7629112 Israel”. In my experience it’s often done like this and you have to dig a bit.
They also bring up the other issue: these services are often expensive as shit. Therapynotes adds $40 onto your subscription for theirs, twofold is like $70/mo, etc and all charge additional user fees on top of that. The exception is headway and I think grow, which are free, but that passes on the cost to the consumer (and imo they’re currently propped up by VC capital, once that runs dry and they’re forced to be “profitable” this could change drastically). In a field where reimbursement rates are relatively high (80-130/hr ish) and yet still most clinicians make 40-60k/yr without benefits because of overheads it’s pretty foolish to add even more monthly costs, though tbf the overwhelming majority of overhead for 99% of clinicians is outrageous commercial real estate pricing
The people I work with don’t relay want an exact transcription, they like how it formats it into a SOAP or other standard way.
They are “supposed” to review each and every note but I see things in the notes I know are incorrect like for example it gets the gender wrong or there are health problems raised which I know the person does not have. Many other times I see things which may be wrong but I can’t for sure tell. It could have happened that way. But it’s a bit weird.
A responsible design trick might actually be to hide intentional errors in the provided text and then have the charts audited at a later date (by a human) to see if those were left in.
I have warned the people I work with that when the VC makes them go public, all the “agreements” made about confidentiality of data will be out the window, to say nothing of the so-far acceptable cost of services.
My hypothesis is that this is a way of getting training data. The business is not really doing ai scribe, it’s convincing HCPs to make patients comfortable allowing ML on their visits. Especially in the set ups where the vendor can then review the note the clinician actually creates and digest that as well. Like how google is a search engine but it’s really a business to profile people for targeted ads.
Also I agree with you that making notes is not purely an administrative afterthought. Documentation is part of the cognitive work. That’s why there are so many forms, tools, scales, flowcharts, acronyms, standards, templates, and such. It isn’t just about recording what happened. It is about structuring your thoughts prior, during and after the visit. And being able to compare them over time.