A group of major US healthcare organizations, including the American Hospital Association (AHA) and the American Medical Association (AMA), has demanded that the Food and Drug Administration (FDA) clearly define legal and technical responsibility for medical devices that integrate generative AI technologies. These demands come as part of responses to a regulatory proposal by the agency aimed at establishing a framework for dealing with these emerging tools.
The Challenge of Responsibility and Transparency
Medical organizations argue that physicians and hospitals do not possess the technical capacity or resources necessary to continuously monitor the performance of complex AI models. The American Hospital Association emphasized that developers are the party best equipped to evaluate and maintain these tools, noting that the lack of transparency in how these models function makes it difficult for care providers to detect software flaws or bias in outputs.
Concerns Regarding Human Oversight
The American Medical Association (AMA) raised a fundamental point regarding the concept of "human-in-the-loop," asserting that the presence of a physician overseeing AI outputs does not necessarily guarantee risk reduction. It explained that over-reliance on these tools could lead to clinical errors, which necessitates evaluating the performance of the human element as well within the risk assessment framework, rather than relying solely on the nature of the device's operation.
What Changes Practically?
- Shifting the Burden of Responsibility: Medical organizations are seeking to protect themselves from legal liability for technical errors they cannot control, maintaining that responsibility should rest with the manufacturing companies.
- Demands for "Nutrition Labels": Some groups, such as WEDI, have proposed the necessity of requiring companies to provide explanatory labels (similar to nutritional value cards) that clarify the tool's functions and potential risks to patients and physicians.
- Developing a Risk Framework: Medical organizations believe that assessing risk based solely on "device activity" is insufficient, and other variables such as the probability of error and the negative outcomes resulting from use must be included.
These discussions come at a time when the US Food and Drug Administration is preparing to issue an official draft guidance on the regulation of generative AI-based medical devices in the coming months, which may put an end to the ongoing debate over legal and technical responsibility in this vital sector.