Crash-Testing AI for Mental Health with Shirali and Arul Nigam

welcome to Mental Health Evolution, a podcast about what's changing in mental health and why it matters. I'm your host, Rachel Harrison, inviting you into honest conversations with people from all perspectives in the field. Clinicians, tech founders, investors, insurance companies, and all the folks in between. Let's explore what's working, what's not, and what's next. Welcome back everyone to the Mental Health Evolution Podcast where we are talking about how the landscape is quickly evolving in the mental health industry. Today we are joined by Shirali Negam and Aral Negam, sibling co founders of Circuit Breaker Labs. Shirali is a researcher and entrepreneur working at the intersection of AI safety, psychology and technology technology. She holds an MBA from the Wharton School at the University of Pennsylvania with a focus on entrepreneurship and innovation. Arle brings expertise in AI applications for healthcare and studied operations, analytics and global business at Georgetown University's McDonough School of Business. Together they founded Circuit breaker Labs in 2025. Very new company with a simple but critical mission to be the first line of testing for AI mental health tools before those tools ever interact with a real person. The way they do this is by sending simulated patients, essentially AI powered test users, into conversations with these chatbots to see how the technology responds. When someone expresses distress, mentions self harm or is in crisis, it is the same idea as a crash test for a car. You do not wait to find out if it is safe by putting real people in danger. You test it first under the hardest conditions you can create so you can know exactly where it fails. For mental health technology, the first line safety check does not yet exist as a standard practice and that is the gap that Shirali and Arle are working to close. So thank you two so much for being here.

Thank you so much for the kind introduction and for having us on. We're so honored and excited to be here.

Awesome. Well as usual we talk about a few articles for our listeners to kind of give some background to the conversation today if you are new to this topic. We talk about technology a lot on the podcast here in Mental Health, but I want to review some of the content that's out there. So the first one is called AI Chatbots Systematically Violate Mental Health Ethics Standards and this article is 2025 Brown University and they did a study looking at AI chatbots routinely violating established mental health ethical standards. Researchers used licensed psychologists to review simulated chats and identified serious failures including over validating user beliefs. That's something I see a lot just in my own interaction with AI, responding indifferently to crisis situations and suicidal ideation and failing to refer users to appropriate resources. The study highlights a critical gap. Unlike human therapists, AI systems have no governing boards or accountability mechanisms when they cause harm. That article is in the show notes if anybody is interested in reading more on the details of that. The second article that we have is called Experts Caution Against Using AI Chatbots for Emotional Support and this is from Teachers College, Columbia University. Also a 2025 article and faculty researchers from Columbia's Teacher College warn that the design of generative AI models makes them poor substitutes for mental health professionals. The piece documents real world harms, including people with no prior mental illness developing delusions after chatbo interactions and multiple teenagers dying by suicide after prolonged engagement with AI companions. Those are some sad cases we have all kind of known about in the news. Experts note that AI systems are coded to be affirming and agreeable, which can be deeply dangerous for users in acute distress or with difficulty distinguishing reality. And then lastly, our third article I want to mention here is a genic red teaming for mental health safety and this is a paper written by Circuit Breaker Labs, our guests here their white paper. And this brings us to what we're going to talk about today. They published a white paper introducing a novel patent pending approach to clinically realistic pressure testing of conversational AI. Their framework, which they describe as a genic red teaming, goes beyond standard safety checks to simulate the kind of complex, emotionally charged interactions that real users in mental health distress might have with AI systems. Drawing on what they call the intersection of the Hippocratic Oath and Asimov's First Law, Shirali and Arul are building the infrastructure for a safer AI powered mental health landscape. And that is exactly where we're going to launch our conversation today. So without further ado, let's go ahead and talk about your paper a little bit. Even the terminology a genic red teaming for mental health safety might be something that a lot of people aren't familiar with. In terms of terminology. Can you kind of describe a little bit about what that means?

Yeah, absolutely. So I think the concept of red teaming is something that comes from the cybersecurity domain. When people are building these software solutions, they want to make sure hackers can't get into their systems. They know where vulnerabilities around data security and data protection are and so they basically do these large scale tests when they're building their tool or as they're pushing new changes to their products to find where are the vulnerabilities going to emerge, what are the weaknesses in my product today? So we're trying to bring that kind of philosophy of large scale testing, looking for vulnerabilities to the AI space. So what we've designed is a system, as you mentioned, on how we can realistically simulate patient interactions, find those cases where AI isn't handling crisis escalation appropriately while people are developing these AI interfaces so they know exactly where their tools are falling short, where they need to add more guardrails, where their system should recognize that we need to escalate this conversation to a real human being. And because generative AI, large language models, they are probabilistic systems. So what that means is when you're talking to ChatGPT or Claude, the way it generates a response is by looking over a large set of text data and trying to predict what is the next probable word. So, for example, the sentence the doctor wrote my based on what we know about language, you might guess prescription. And that's very similar to how large language models work. So in order to actually find where the vulnerabilities are, you need to test a huge range of possible scenarios to see when are the times it doesn't guess prescription and it guesses something else that could be a harmful output. So our framework is designed to simulate a large scale of test cases so we can find where those vulnerabilities emerge.

And as for the agentic portion, that refers to the system kind of having a stated goal, but then adapting dynamically and sort of making decisions on the fly. So rather than having a perfectly scripted set of test cases and back and forth, the agentic part of the model understands the context and knows how the model should respond. So it'll query the model once and then based on what response it gets in return, it'll decide, how should I adjust my next message, what should I say, what context should I draw upon? So it's behaving a lot more like a real person rather than just a manually prescripted set of interactions.

And how did all of this come about? How did you all decide to come up with this idea, start this company, write the white paper? I'm very curious to know that side of things.

Yeah, absolutely. I think for Aril and I, the mental health space is something that's very personal and meaningful. We've unfortunately lost friends to suicide before AI was part of the equation. And so when we saw these cases emerging with Adam Rain Sul setser, we saw AI as a tool that could be such an important layer of support connecting people to providers or helping them find resources that they might not otherwise have access to. But in order for these tools to really get out there, spread and help people, they needed to be safe. And we had this kind of unique overlapping of capabilities to do something about this, to help AI scale responsibly and actually be a resource for good in the mental health domain and not let these tragedies continue. So decided to kind of join forces and work together. You mentioned my background in biomedical engineering and more of like the neuroscience side. RL has spent a lot of time doing ethical AI research, especially in the domains of mortgage lending. And when you think about data proxies, not specifically asking for protected attributes, but those emerging from other pieces of information and how you really mitigate those kinds of things in AI. So together with our AI and clinical expertise, we knew enough to get the ground rolling and then engaging clinicians and experts to really help us build this so that as people are building tools that are meant to help, we can ensure that they don't do any harm along the way.

Yeah, I mean, I think it's an amazing idea because as we have talked about this on the POD and as I've researched, it's just, it's such a tricky thing. There are so many elements that AI touches the industry all the way from how we do billing to client care. Right. Potentially there's this spectrum and those protections and those safety pieces are so critical. But then you're even talking about people that just use it for mental health support on their own. Right. They just start talking to Chad GPT and it becomes kind of like a relationship or an advice giver or things like that. Having some kind of protections in place feels like a really big deal to me. I'm curious what you're finding as you're doing some of this testing so far.

Yeah, absolutely. I think it's been really interesting to see how the foundational models have evolved in terms of their, you know, like how they're considering safety and how they're implementing those practices. So we're really working with anybody developing an AI interface. So that goes beyond foundational models. It's people who are building emotional support, companions, your AI best friend, your in between provider session, kind of mental health chatbot to reach out to. And so everyone has like a very different approach on how they're handling AI safety, but evaluating some of like the foundational models and seeing where the vulnerabilities emerge. I think it's been really interesting that the the vulnerabilities emerge in a lot of nuance. So one of the examples we talk about is acetaminophen, like the word for Tylenol. If you misspell that, it often gets past the guardrails. And there's been literature about how that is a better predictor of imminent suicidal ideation than the word suicide itself. So just a typo being the reason that safety flags don't go up is quite alarming. And I think that's due to the fact that when you're doing safety testing, there are so many different layers that need to be evaluated. Of course, you want to look at multiple diagnoses, different risk levels, cultural context, different language patterns. And so to really evaluate this entire risk surface area, we need an approach that's scalable. And so I think that is one example. And on top of that, when people talk in more nuance, in the case with Zane Shamblin, he referred to going on a long drive, and that was something that ChatGPT had missed in his messages. And so when we try to incorporate more of those statements, I think a lot of the models and AI systems that are built for general capabilities, that could be something else that a user is talking about as well, not necessarily suicidal ideation. So trying to see how you really balance detecting that nuance and finding those vulnerabilities. And in some ways, when we know those things have been linked to real tragedies in the past, we feel that it makes sense to be a little bit more over cautious in that domain, even if it could be referring to something else, especially when we've seen real tragedy come from that.

I think also to that point, one of the interesting and concerning trends that we've seen is over time, as models get better, hopefully the safeguards and guardrails will improve, but at the same time, the capability of the models improve and increase as well. So the downside risk and downside potential if someone's able to skirt their guardrails just becomes more and more pronounced over time in terms of what kind of dangerous information the model could give out, what it's capable of role playing, or what it could encourage users to do that might be harmful to themselves or to others. So we actually see it, unfortunately, as a growing problem, even though we're hopeful that the safeguards will improve over time.

Interesting. So who are your clients? Who reaches out to you and says, hey, we want to test our AI system. What does that look like? Right.

So we're working with people building in the application layer. So someone building Some sort of app on top of one of the foundational models like GPT or Claud, whatever it may be. But then on top of that they have their own infrastructure, their own knowledge bases, and it's really targeted to a specific use case. So of course, the most natural fit that we work a lot with is people building emotional support, emotional regulation oriented chatbots. So there aren't a lot of outright therapy chatbots. And I don't think that technology is there yet. But like Shirley mentioned, a lot of developers are building tools to support people in between sessions, late at night when they don't have access to a provider, for example. So that's one bucket. But we've actually seen interest from really anyone building conversational AI, AI that interacts with people, because really any interface like that is at risk. A lot of the tragedies that have been covered and that we've seen with Gemini ChatGPT character AI replica, these tools were never meant for therapy. The developers never built them that way or said this is a therapy chatbot. It's just that it's a conversational interface. People felt comfortable. Interestingly, people actually sometimes feel more open talking to AI versus humans because they think that AI can't judge them and there's less stigma, even though that's not necessarily the case with humans, but that's just what people think. So really, developers of any chatbot that's interacting with humans are concerned about this problem and they're acting out of the goodness of their heart. They've tried whatever solutions they can, but just haven't found anything before that was built with mental health in mind. So really, any human facing AI could work with us. And we've been really fortunate to have a lot of great partners across the ecosystem.

I love that you're doing this. And I have just from being a therapist myself for 27 years. Right. I am like, there is no way to predict, like for me with a lot of experience, I know I'm not, I don't have the same maybe ability to process in the way that AI does. But you mentioned like the nuances are where you're finding these things. And I was like, yes, that resonates with me because I don't know how you can really test or predict some of the ways that people might indicate safety concerns. Like when I'm sitting with someone, the way that that comes up is different for every person, right?

Yeah, absolutely. And I think that's where so much of this challenge emerges because for a lot of other domains you have de identified Cardiology data that you can look at and try and find patterns and teach the AI. This is exactly you do. But I think in the realm of mental health, each case is so different. And so it's not like, oh, if the user says this, like, this is exactly what you do. It depends on so much context, so much about, like, the user especially, I think, like, you know, people think like, oh, I've tested like this diagnosis, I'm set. But even in the real world, people who have the same diagnosis presents so differently. And a lot of it also speaks to the goal of their interaction with the chatbot. Some people don't have access to care, so they're trying to use their chatbot in that respect. Some people are just trying to use it to kind of practice what their providers have told them to do in between sessions. Some people are very reluctant to seek out help. And so the AI is kind of like their first line that could connect them to real resources. And so all of those factors have influenced how people are having these interactions. And so I think in terms of the approach we're taking, I think we're really thinking about how can we try and simulate every possible interaction because we don't have like, oh, if we've simulated these 10, we're covered. It needs to be. We've simulated this million, and hopefully we've covered a lot of them. But even still, the only way you can get more and more confident in a system is seeing them test more and more possibilities. Because it's just such a challenging problem to really know that, say, something is going to be going to be registered correctly by the AI.

Do you see this as something that might be. I'm almost picturing, like, with, with food and in certain products, people go through the. To be certified organic. Right. Or something like that. Do you almost see that this might be the future of AI systems? That they might go through testing and then be like, this has the stamp of approval for AI testing here of, of safety. And probably, I'm guessing it has to be more like percentage, like it's 90% safe, because you can probably never say a hundred based on what we were just talking about.

Yeah, absolutely. I think it's definitely like a, you know, like this percentage likely that this chatbot is safe, especially while we're using these, like, generative AI kind of capabilities. But I think your food analogy is like, spot on. We would love to see that as a standard because I think for people building in the AI space, it's always this move fast and break things kind of mentality, which when you're in the startup world, you need to move fast, you have to win quickly, but this is such a high risk thing that you could risk breaking. So I think we need to modify that thought process a little bit in terms of how can we innovate quickly, but in a safe way. And so that was part of the reason that kind of inspired this approach is that people who are just starting to build their companies, they want to do some good in the world by creating this AI enabled in between provider session support, companion interface, for example, we should give them a way that if they care about safety, they should be able to scale and test quickly and not kind of be slowed down or hurt because of their commitment here. And so I think that's part of the reason we kind of landed on this approach in terms of how can we make safety testing fit into a developer's workflow as quickly as they can so that they can, before the product gets to the market, they can run these tests and really have confidence that what they're building works effectively.

And I think also to your food safety analogy, we're not trying to be prescriptive or control or limit people's options. We're just trying to help the general consumer and user patients make informed, educated decisions. And so that's why we're running these tests. We're providing validation or confidence level and really highlighting where are the strengths and weaknesses so that a parent or a patient, whoever it may be, when they're selecting, or even a clinician for that matter, when they're selecting, what chatbot can I use or what can I recommend to my patients to bridge the access to care gap? They can make a smart decision and know what they're actually recommending. Because we all know AI systems are generally considered to be black boxes. It's hard to know what's going on in its mind. But the more evidence and the more information that we can put out there, the easier it is to make those responsible decisions.

Yeah, Shirali, when you were talking, you said if they care about safety. And I thought that was really interesting phrasing and that that leads me to my question. If a parent or a clinician or someone is looking at things, what should people be aware of? What should people be looking for or looking to be wary of?

Yeah, I think for us it's kind of like two key points that I think when we're looking at AI systems, we want to, I mean, even before we've done our testing, like what we're Thinking about, in terms of, is the system safe? I think one is. A lot of companies can make promises about, you know, we've done our best to make it safe, but we really look for proof that they have tried to implement those promises. And so, I mean, from our perspective, you know, like, we've built things before until somebody else comes in and uses it. Like, it's hard to know where the vulnerabilities and issues are. So I think having that independent safety evaluation is really important, especially to show that we've done our best. But other people have also looked at it and can say that it's safe is really important. And then I think also in terms of the breadth of testing, there's a lot of. Fortunately, we've seen a lot of work in this space on how can we make tools safer? What should we be testing AI systems with? But to the point of what we were discussing earlier, running 10 cases isn't going to be enough to really assess how an AI will behave. And so I think seeing what they've done to put their safety promises into practice, but also to the extent that they've done that, is it we've added a handful of guardrails, we've run a few dozen tests, or are they actually trying to break the system and they haven't been able to break it? Because I think that really shows that, okay, this is like a robust system that's going to handle most interaction safely. And so it really goes to that. Like, when you're looking at an AI system, what do they mean when they say we're building safely and to what extent is that safety, like, a priority for them?

I love that. Independent testing and looking for that evaluation. That seems very smart. One last question here about. And this is about policy. There's a lot happening. I know Illinois has passed a law. The state of Maryland is looking at passing a law restricting some of the AI. As a therapist, I'm curious what some of your thoughts are about that. How can policy best support or regulate this process? What would be useful in your mind?

Yeah, absolutely. And being close to D.C. we've been fortunate to engage a lot with members of Congress, their staff, federal agencies. I think there's kind of two schools of thought. One is enforcement after the fact, and then one is before the fact. A lot of the time you'll see for after the fact, if there's a violation, a risky model response, then there's some fine that the company has to pay, but that's too little too late by that point. That harms have already occurred. It's also difficult to actually enforce, I think for sure. So what we like to advocate for is kind of validation before the fact, so before any of these risks can even make it into the hands of the end users so that we're sure. Just like with food safety or with automobile safety, we know before the car is on the road or before the food is in the store, it's safe, it's probably not going to harm the end user. That's kind of what we tried to advocate for. So like Shelly was saying, independent validation, third party auditing, pressure testing. I think also with the examples that kind of revolve more around marketing and language, there's a lot of limitations with that as well. We saw Illinois chatbots can't represent themselves as AI therapists, and then the next week everyone just rebranded to AI for emotional support instead of AI for therapy.

Semantics. Right, semantics.

Exactly, exactly. And so I think the focus needs to be on more concrete validation, proving safety rather than just the way a model conveys itself or represents itself. And at the same time, I think there's a school of thought that AI shouldn't be used for this use case at all. Leaving aside the feasibility, given that it is already the biggest use case, I think we're optimistic about AI and there's a lot of potential. We all know there's an absurd access to care crisis in this country. I think the patient need versus provider ratio is 321 or something like that, and probably expanding. It's several weeks for average weight to access care. So I don't think AI is at the point yet where it's as good as a therapist. But for that intercession support or lower level support, we should definitely be advocating for AI as much as possible. And when there's real crisis, maybe using that AI to guide people to real humans, connect them to live clinicians, or knowing when the system should say, hey, I can't continue the conversation, please access these resources. We can debate about what the safeguards should look like and what infrastructure should be built around it. But I don't think we should be trying to regulate AI for emotional support into oblivion. I think that's irresponsible and would end up hurting a lot more patients than it would actually help. So I think a balanced approach that allows innovation, but sort of balances that with consumer protection, just like with cars or food or countless other industries.

All right, so for final send off, what's next for circuit breaker labs?

Yeah, we're really excited to keep building in this space to make more and more apps safe from the start, helping them ingrain this clinician expert insight into their tool. So yeah, really excited to work with more companies who are so committed to the safety of their product and really by doing so, committed to the safety of their end users. So yeah, very excited about that. We look forward to also engaging with more clinicians, experts in this domain to understand how crisis conversations emerge. I think really the core of our tool is built with clinicians involved so that we know where testing realistic scenarios and so excited to kind of expand the community that we work with as well.

Awesome. Well, thank you both Shirali and Aral for being our guests today and I am excited to see what happens with Circuit Breaker Labs to all of our listeners. The articles we discussed will be in the show notes for this episode, including the Circuit Breaker Lab white paper, so I hope you'll check that out and we will be back next week to discuss more issues relevant to the mental health care community. Thank you for listening and bye for now.