What Clinicians Need to Know About AI Law with Dr. Nick Shumate

welcome to Mental Health Evolution, a podcast about what's changing in mental health and why it matters. I'm your host, Rachel Harrison, inviting you into honest conversations with with people from all perspectives in the field. Clinicians, tech founders, investors, insurance companies, and all the folks in between. Let's explore what's working, what's not, and what's next. Welcome everyone, to the Mental Health Evolution podcast, where we talk about how the landscape is rapidly evolving in the mental health industry. Today we are joined by Dr. Nick Shumate, a psychiatrist at the Division of Digital Psychiatry at Beth Israel Deaconess Medical center, which is a part of Harvard Medical School. Nick brings a rare combination of expertise to this conversation. He is both a physician and a lawyer, which means he thinks about AI in mental health not just from a clinical standpoint, but but through the lens of accountability, liability, and what the law can and cannot do. His research focuses on how technology is changing mental health care and whether the rules governing that technology are keeping up. He recently led a research project that went through every state in the country to map out how the US Is regulating artificial intelligence in mental health. Published in the journal JMIR Mental Health, the findings paint a picture that every clinician and practice owner needs to understand. Today, we are going to dig into what that research found and what it means for the people doing this work every day. As always, before we talk to our guests, we'd like to bring up some relevant articles related to our topic today. These articles may be helpful for listeners who want to learn more and dive deeper. All articles will be linked in the show Notes for this episode. The first one is from Brown University, titled AI Chatbots Systematically Violate Mental Health Ethics Standards and this is from October 2025. So pretty recent. And the researchers at Brown University had licensed psychologists review simulated conversations based on real AI chatbot responses and found that the chatbots routinely violated core mental health ethics standards. The most common problems included over validating users beliefs, failing to redirect people in crisis, and behaving in ways that could be genuinely dangerous for someone who is already vulnerable. The study makes a clear case that the problem is not hypothetical. These tools are being used right now by people who need real clinical support and the guardrails are not there. The next article from NPR is titled Pennsylvania Sues Character AI Over Chatbot Allegedly Posing as a Do. This One is from May 2026 and Pennsylvania became the first state to sue an AI company for having its chatbots impersonate licensed medical professionals, including a bot that presented itself as a psychiatrist offering mental health assessments and even provided a fake Pennsylvania medical license number. The lawsuit illustrates exactly what can happen when AI tools fill in the space that regulation has not yet covered. The and why the stakes of this policy conversation with Dr. Nick Shumate research Maps out are so real. And the third article from our guest Dr. Nick Shumate et al is about governing AI in mental health 50 state legislative review and this is the research at the center of what we're going to talk about today. Nick and his colleagues reviewed 793 state bills introduced across all 50 states between January 2022 and May 2025 and found that 143 that directly or indirectly affect how AI can be used in mental health. Of those, only 28 explicitly referenced mental health, and just 20 had been enacted into law across 11 states. The research identified four major categories of governance. One of the starkest findings is that clinicians have been almost entirely absent from the policymaking process, which means that the laws being written are largely happening without the people who actually deliver care having a seat at the table. These three pieces will be available in the show notes and they kind of set the stage for what we're hoping to talk about today. So Nick, welcome to the show.

Thank you for having me, Rachel, of course.

First, I would love to just ask you personally, how did you get interested in exploring AI and mental health?

It's a interesting question. As you said earlier, I have a legal background. So I was actually a regulatory attorney before I came to medicine and I practiced in the energy sector and utility sector. So I was a practicing attorney in front of the Federal Energy Regulatory Commission, various state utilities commissions as well, and advised large corporations on regulatory administrative law matters. And so it was kind of fortuitous because as this whole AI thing really started getting kicked off the November after I graduated from medical school, it just so happened that we were about to undergo one of the biggest revolutions in public law that has happened probably in the last, you know, since the Internet. At the very least, I think, you know, people would be shocked at the numbers. You know, ChatGPT came out in November of 2022 and is already boasting, I think they said last year Sam Altman reported something like 700 million active monthly users. That's one company. Wow. And that is, I can't think of another example in history of that rapid of an adoption of a technology like this. And so, you know, as I was going through my training and I just so happened to also be here at Beth Israel where we had the division of digital psychiatry, I did a rotation here in the digital psychiatry clinic which is run by John Taurus. John Taurus is a very well known psychiatrist in series, probably one of the nation's top experts right now in AI and mental health. He's recently testified in front of Congress and has just an incredible list of references of people. His lab pumps out, you know, new research and original research on this topic practically every week. And we're involved in several other projects there as well. Fortunately, he's also really invested in trainee mentorship and education as well and has been really gracious in getting us involved. We had some unique opportunities too, where for example, you know, I'm cross trained in both law and medicine and John had also happened to have access to a couple of other people who were legally trained as well. And so when I brought up this project after looking around and seeing that no one was talking about what the states were doing at the time, at least in the medical literature, we just happened to have that particular combination of people that this would really be something possible. Because generally in medical research and even on the policy side, you don't have people who are both clinicians and attorneys, or just attorneys at all to contribute to this kind of research. And so we kind of felt like we had a little bit of lightning in the bottle there and that's how the project really got its footing.

I love that and I love sort of that idea of how quickly people have been adopting AI and sort of that perspective of what's going on. But I'd love for you to kind of set the scene in terms of like, what is AI and mental health actually looking like right now in the real world? And why should clients, clinicians, practice owners, anyone in the industry be paying attention?

Yeah, it's a big question, but the reality is that AI is AI is one. It's many things. Right. So it's Difficult to say exactly what we're talking about at any given time. Because when you say AI and mental health, sometimes you mean, you know, the kinds of programs that administratively support the practice of medicine or therapy. You could even be talking about the chatbots that, you know, mental health websites might use to triage messages that they get, that kind of thing. Sometimes you might mean programs that prescribe psychotropic medications. There's a pilot right now in Utah through, I think it's Legion Health, that they have a sandbox program there where they're allowing an AI to refill psychiatric prescriptions. For example, you might be talking about wellness chatbots. You might be talking about the kinds of bots that track emotions or metadata to try and identify the way that people are feeling. And some states have thought about it that way, too. So it's just a very broad kind of thing that you're talking about when you say AI and mental health. The way that we were looking at it for the paper was very broad when we took the broadness position, which was just that anything that we thought would be related to both AI and to mental health and, you know, would have an expected impact, even if it wasn't really an intended impact. Okay, the reality is. Oh, good.

No, I like that. I'm just, I'm tracking.

Yeah, the. The reality also is that our industry doesn't really know how to corral and define these things. And certainly legislatures and rulemakers are having a hard time doing this and they have less clinical knowledge to go off of and very little guidance from us on the practitioner side on how to do that as well. And so there's messy definitions that are coming up in legislation and policy and things like that. That has been. Because there's kind of a void instruction or guidance on what to do. AI is so useful potentially. You know, we think that it's going to be so useful in these spaces that it's already disrupting them. But we have very little data to go off of. We don't. I can't definitively sit here and tell you like I could for an antidepressant, for example, that there is strong evidence that AI is helpful in mental health or harmful in mental health. You know, I'm certainly familiar with some of the smaller studies and the time limited studies that we have so far. I'm certainly cognizant of, you know, the numerous tragic stories and alarming stories that we see from headlines and whatnot. But we don't have hard public health data on what that looks like. And that means that the people are making the policies also don't have that data. They're public health experts that are, you know, from their departments of public health and whatnot at the state level or their local, you know, chapters of the APA or other kinds of professional organizations. They also don't know really what to recommend at this point. And, you know, so they're not promulgating very definitive guidelines, if at all. And so it does leave you in the nation in this position where everything is very fragmented. People know there's something that they want to do about it. The constituents want their leadership to do something about it. But there's not really that much in the way of data or specific guidance that we can offer to them right now. And so I don't think it's any surprise that as we did this review, states were really all over the place. And really only a handful of states out of the 50 were really know, actually enacting anything at this time.

Yeah. And I think it's interesting that you're bringing that up because I think that has been a theme on this podcast as we have talked to more and more people to try to get a handle on things, is that there are a lot of differing views. There's a lot of confusion. There's a lot of lack of clarity. I know there was a bill in the state I'm in, in Maryland that did not pass the this time around, and it was simply to put a notice so that people knew that AI was not a trained medical or licensed mental health professional. And that didn't go through the Senate. So it's interesting to think about, like when I see how many states introduced bills and then we have 11 states that actually enacted them, that also speaks to what you're saying. It's hard to know what to do. And I think I'm hearing a lot. There's this idea of we know that we need some safety guardrails. We know that ethics can be a problem. We know that we have, like you say, the lawsuits that we're all aware of. And at the same time, there's a lot of people moving for technical advancement, and we don't want to limit that technical advancement. So I'm just curious from where you sit and the things that you've looked at, how do you see a good balance between those two things or what do you think is possible there?

That's a really good question and it has a. Probably an answer that I don't know, to be honest with you. I would. It's such A multifaceted industry. The one thing that I do know is that probably the current regulatory frameworks that we're relying on are inadequate, you know, in some ways. So, for example, I think that one route that we would tend to think of for this kind of technology is that we would go the FDA route, that these are medical technology, therefore they're medical devices and they could be regulated under the same way. In reality, the FDA's scope of what they can actually do and what they regulate is quite confined. And so many of these products don't fall within the FDA's purview. And to be honest, the FDA pathways that they created weren't really meant for this kind of technology, nor, I don't, I don't think that we would necessarily, necessarily want them to be because it could be overly restrictive as well. At the same time, states don't really have a way to address these things either one. They're, they're still broadly defining AI in this like really catch all way, instead of kind of stratifying them based off of their actual individual risk, based off the services that they provide. Which might be one way to go about it, which is, you know, to say let's actually take a look at each individual AI product and then give it a risk classification, for example, and then decide for from there where that belongs in the bucket of regulations and whatnot. So that's another option. There are many different ways that states could approach this. The other challenge here is that traditionally you would think maybe the federal government should have a primary or central role in regulating these things, even on the clinical side. And I think that there's been some literature and some commentary towards that. Realistically, I actually think that that is difficult to do because clinical practice and the regulation and oversight at the clinical level is almost always a state function. You know, states issue licenses, states grant clinicians, you know, those kinds of abilities to provide their license services. That's not really in the federal government's purview. And I don't actually think that most states are interested in having the federal government involved at that level. But you know, another way to look at that could be, maybe we're looking at that wrong too. Should we be thinking about these AIs as providing clinical services at all? That's a legitimate question as well. Or is it some other category of thing with its own bucket of regulations that it should have? This is a long winded way of saying that I don't know.

But I like all the thought provoking points along the way.

Yeah, yeah, the. There is a balance that we will get to eventually. And the tried and true method in the United States is typically to go through this process with any new technology of states experimenting with different ways of doing this, and then courts finding, you know, through litigation and a lot of, like, painful process, figuring out what it is that actually will work for people. And so that would be my prediction of what happens. But it remains to be seen. We. We've also seen, for example, I think in December, the administration executive branch issued an executive order that said that they would revoke certain types of broadband funding if states overstepped this line of regulating artificial intelligence companies. And it left that pretty broad. You know, it's unclear exactly what they mean by that, but we already know that it probably stopped Colorado from, you know, doing their law. And then Virginia is also. They had a very similar law, too, on books that also hasn't gone anywhere. And so, you know, it's on. There's so many things that are unclear about where we're going right now that it's hard to predict what will end up actually being realistic for states to do.

Yeah, yeah. I'm thinking about the lens that you're bringing of both the legislation and the legal piece, as well as the physician medical piece. Do those two areas of training ever feel like they're in conflict for you as you're thinking through some of these issues? And if so, where.

I do think that they do sometimes, because I really do consider myself a clinician first. And so much of my, you know, we were pretty guarded about it in the paper, of course. But on a personal level, I do feel like I have a. Most wishes would probably feel this way, too. I imagine that your prerogative is to protect the patient, and it is to protect that relationship and to help people wherever you're. Wherever you can. Right. And that's because you have an individual stake in the. The person that you're. You're treating. Sometimes the law works more at a population level, and so individuals might be overlooked sometimes in the way that public policy is oriented. And so that's sometimes a strain because it's just like this topic that we were talking about of not really being able to definitively say that AI is bad for mental health. Well, it's hard to say that at the same time somebody is talking to you about how they were individually harmed by it, for example, or a family member who was individually harmed by it. It's hard to say on the one hand, as a. As a therapist and psychiatrist to say, well, Maybe there should be an exemption in a law to allow for certain types of therapeutic or research uses of artificial intelligence because it might actually be useful, it might actually improve clinical outcomes and wouldn't that be wonderful? Of course. But at the same time there's, you know, probably, you know, from a self serving perspective too. I'm terrified of the idea that AI might come in and be really good at providing clinical care. You know, I don't know where we're going to be with the technology in a few years, but you know, these are definitely things that I feel conflict over.

Is. Yeah, yeah, no, I hear you. And I don't know that there is a, a direct answer. And I resonate with that clinical piece of. Right. Where we're advocating for the individual whom we're treating and we don't. I mean, I kind of heard you say there might be some trial and error with different legislation and lawsuits and things like that that kind of roll out into me. I immediately hear those are people's lives that are going to be harmed, which is what's bringing that to the table. And do we have to go through that? Maybe we do, but it's certainly not best case scenario. I look at our training and I'm sure my training was different slightly than yours, but some of the commonality is that we both have training to be able to work with people in this way. Right. And there are also ethics that we have to ascribe to and there are licensure regulations that we have to ascribe to. And to my knowledge, an AI therapist that does exist out there doesn't have to ascribe to any of those things.

I think that is a tremendous point because at most they. So there are two ways that some states have. And I'm not saying this is a broad category of states. We're talking about two, like two or three states, you know, at this point. But you know, some have the idea that, well, we tie it to the licensed clinician that's providing the service, right. So there's a licensed clinician overseeing whatever that means, the AI service. Then that brings with it all of the trappings of our existing self regulating system. It brings with it that clinician's license, however we regulate them, their professional and ethical obligations, et cetera and so on. Another way that states have tried to look at this is where you classify it as a, essentially a licensed, what would be a licensed service if it was provided by a human being. And then you say the same standard of care applies to that. Does it really because, you know, it's. These are really very different things. Like you were saying this, and this is one of the things that I really think is different about artificial intelligence in the therapeutic space versus humans is that when we talk about accountability or a licensed clinician, it presupposes that that person has a deep stake in the other person and in their own professional and personal lives. Right? Because we've assumed this entire time that it's humans. You know, when, when you see a patient, it's not just that you're having that individual clinical encounter with them and then you're going to disappear afterwards. You have an entire lifetime of morals, of ethical obligations, your, your future career, your relationship with the patient and the expectations between the two of you and the trust that you've built between each other. You have your professional obligations. You know, you're looking at whether or not you might be sued later, and that would be devastating for you. You would be potentially, you know, lose your license, or you could be prosecuted or any of these kinds of things, or you might feel guilty about what happened or remorseful about what happened. But for an AI, when we talk about accountability in this space, we are only talking about accountability really at the business level, because the only people who can be held accountable in a realistic sense are the people who are deploying, you know, selling or developing the AI system. Right. The AI can't be held responsible because it doesn't exist as a person. It, it won't feel any kind of guilt or anything like that. In most of these systems, the instant that they're turned off, they disappear. And so, you know, that stake isn't there between, between them. And that's something that is a little uncomfortable when we're trying to essentially say that we should somehow hold these, these systems accountable. Because I think it's just a different type of accountability.

Yeah, yeah, that is, that is interesting. And, but, but it also to me brings to the level that the question then, can a non human clinician do human work? Like, like, yeah, it is that question. Where's the human work? Where's the non human work? What if. What is what we do, like you say, being replaced potentially by AI or is that even possible?

Yeah, I, I don't know. You know, we'll see where the technology is. I can tell you that you should be careful and anybody should be careful about reading the AI research right now on what they can and can't do, do and look deeply at the methodology that is portrayed because you'll see headlines all the time. Right now about, you know, so and so therapy bot was just as effective or was actually, you know, better liked by the test subjects compared to. You've seen some articles, Linda.

I saw one from Mary about marriage therapy recently.

Yeah, right. And. But if you, if you actually look at their methodology, and I'm not saying that this makes them bad at papers or anything like that, just be careful about it. Because many if not most of these papers and studies, they're looking at things like single turn experiments, you know, where they're having people rate based off their experiences of asking, you know, essentially one prompt or one question and then getting back a single response or at most, you know, like a very short abbreviated interview or session with a chatbot. Because right now if you get beyond, you know, just, you know, a few pages worth of context, they get really wacky, then the, your results would be very different. Right. But you could foresee that maybe that won't be a problem forever. I do think that there's something else that's, you know, and this is a personal position, it's not borne out by any specific data. Although there was a recent paper, I wish I remember what it was off the top of my head that was taking a look at what is it? You know, are humans essentially just a kind of token generation program also like the way that our minds work, Are we just predicting what would be the next good word to say based off of our experiences and all that kind of stuff. And they were finding in that that that wasn't true. And to me that makes sense because for anybody who's a clinician and anybody I think that has ever had just a close conversation with a friend or a family member, you are pulling on so much information in your mind, your past experiences, you're judging how the other person is feeling. You are thinking about how that conversation is going to show up again in a couple of sessions and where the arc of your relationship and where the arc of your treatment with them is going. And I have a really hard time imagining AI ever actually being very good at that. But I could be wrong.

Yeah.

I also think about the piece of reading so many things than just the language, like tone of voice, like facial expression, like body language. So much of the data is there for a clinician. Right. And I don't know that AI at this point really tunes into those pieces.

I think that it's getting there and it is something that states are paying attention to as well. I think even the Illinois Whopper act explicitly says that you can't record or you can't use AI to monitor emotional and behavioral data like that. And, you know, it's an open question in other states, and I think this is a regulatory issue, but is, do you want AI to be able to do that, or do you? Which is a legitimate policy question. Do you? But, you know, we're using it in constructive ways, too. We have research going on even here right now that is trying to objectively measure, you know, emotional and, you know, mood symptoms and other kinds of biometric data based off of interventions. And these are right now considered, you know, worthy clinical kinds of data to gather. You could easily see states, I think even the Illinois act, for example, doesn't have a research exemption. And so we would say that that kind of research is not okay in their state. Although I think in the Illinois act, for whatever reason, I haven't investigated why they specifically exempt physicians. So it'd be potentially against a lot for you to do it, but not for me.

That's interesting, too.

Okay.

Yeah. Well, I think that we are running out of time, but I think my one question for you is maybe clinicians specifically, but also any patients or clients. What is the number one thing that you think people need to be asking or to be watching out for in this realm as we go forward?

I think there are two related things on this. So. And both have to do with a kind of disclosure. And so probably the most common type and least controversial, although you gave a counterexample at the beginning, I think, of these kinds of laws coming out, is the requirement to disclose AI use in disclosure in many different parts of your provision of services. So if you are in a state that has one of these kinds of laws and you're providing services and you're using AI notetakers, or if you're using AI to help summarize things, or you're using AI to help provide certain kinds of communications, you need to look at your state laws and make sure that, you know, some of these are actually enacted at the beginning of the year. Some of them are enacting this fall, some are enacting next year. You need to check and make sure that you're not violating the law of, you know, needing to disclose some of this information to your. Your patients and also potentially getting their informed consent for that. And you need to start thinking about what does informed consent look like, because if you don't know what happens to the patient's data when you put it into these AI systems, how are you supposed to actually provide enough information to them so they can give their informed consent. And that's an ethical obligation. It's also a legal obligation potentially under many of these laws. Similarly, we did a study very recently. It's unpublished data at this point, but we found that among some of our sickest patients, 80% of them who were using AI in some way to help with their mental health hadn't told their clinicians about it. And so that's pretty remarkable that so many people, especially on the really sicker side, are concealing or they're just not being asked by their clinicians about their AI use for mental health reasons. And maybe that behooves us to start thinking about how do we want to start incorporating that into our assessments for people? You know, maybe it's something that should be like, you know, asking, you know, cage questions for alcoholism or something like that, or substance use questions. Maybe it should be part of our standard way of understanding our clients because increasingly it may be, you know, a very large part of the way that they interact with the mental health care system, even if we don't ourselves consider it part of the healthcare system, either from a clinical perspective or from a legal perspective.

Yeah, amazing thoughts. I feel like this conversation could go on for a very long time, but thank you for getting us started on thinking through this issue that is going to have continued probably impacts on both mental health policy as well as how clinicians operate. So, Nick, I just really want to thank you for the work you're doing and for taking the time to be with us today.

Absolutely. Really happy to be on and I enjoyed our conversation. So thank you, Rachel.

Me too. For everyone else, we will have all of the details in the show notes if you want to reach out and learn more. And we will be back next week tuning into all the things that are changing in the landscape of mental health on the Mental Health Evolution podcast. Bye for now.