The US Food and Drug Administration (FDA) is moving toward establishing a new regulatory framework for generative artificial intelligence used in medical devices, after receiving more than 500 comments from stakeholders in the health sector, including medical device companies, physician groups, and patient organizations. This step comes in response to a discussion paper issued by the administration last August, in which it proposed evaluating generative AI-based devices through a framework inspired by physician credentialing, with a focus on continuous monitoring after the product is introduced to the market.
Divergent Views on Regulation
The comments revealed a clear division in perspectives on how to handle this technology:
- Industry Sector: Lobbying groups such as AdvaMed urged the administration to build upon existing regulatory frameworks rather than inventing a new framework specific to generative AI, emphasizing that the inclusion of this technology should not automatically impose additional proficiency evaluations.
- Medical Community: Cardiologists in the Society for Cardiovascular Angiography and Interventions (SCAI) stressed the necessity of making a precise distinction between decision support tools and actual decision-making tools, warning of the blurry line created by generative AI.
- Patients and Consumers: The organization Generation Patient demanded enhanced post-market monitoring, warning that over-reliance on subsequent monitoring rather than rigorous pre-market testing could shift the burden of risk onto patients.
Technical and Security Challenges
Research laboratories, such as the Brainstorm Lab at Stanford University, raised fundamental technical concerns, with Dr. Nina Vasan pointing out that generative AI models may exhibit safe behavior initially, but deteriorate as the conversation lengthens. She also suggested the need for a separate evaluation system for failures with severe consequences, such as missing suicide risks or misdiagnosing critical cases, noting that a model's error repeats across all patients and locations, unlike a human doctor's error, which remains local.
What This Means for Users
The importance of these discussions lies in determining legal and technical responsibility for device safety. While companies like Oura are seeking exemptions for devices that provide general wellness information, patient advocates are demanding that chat tools mimicking emotional relationships not be exempt from strict oversight, given their increasing use in providing mental health advice. The question remains open as to how the administration will balance encouraging medical innovation with ensuring patient safety in the face of technology whose outputs constantly change.