Are providers too trusting of AI's advice?
Physicians may sometimes be too trusting of imaging reads completed by artificial intelligence, especially when AI provides more specific explanations to back its interpretations.
A new analysis in the journal Radiology highlights issues with AI’s influence on providers tasked with interpreting chest X-rays. The study revealed that the way AI systems justify their interpretations of an image has a significant impact on how much providers trust said systems’ conclusions. In some cases, this can lead physicians to confidently accept AI suggestions, even when they are not correct.
With the number of AI applications approved by the U.S. Food and Drug Administration expected to reach 1,000 by the end of 2024, it is more important than ever to determine exactly how the algorithms implemented into workflows affect provider performance, as this inevitably impacts patient outcomes, authors of the new paper suggest.
“A gap between AI proof-of-concept and its real-world clinical use has emerged. To bridge this gap, fostering appropriate trust in AI advice is paramount,” senior author Paul H. Yi, MD, the director of intelligent imaging informatics and associate member in the Department of Radiology at St. Jude Children’s Research Hospital in Memphis, cautions.
To determine how AI explanations impact providers’ trust and performance, researchers had 220 physicians (including 132 radiologists) interpret chest X-rays using AI assistance. The AI support tool provided two different types of explanations to back its suggestions—a local explanation guided providers to a specific area of interest, while global explanations provided more general reasoning and used similar images from prior exams to explain its conclusions.
For each case, providers were given patient history, their imaging and the AI tool’s suggestion, which also included either a local or global explanation. Some of the diagnoses provided by AI were correct, some were incorrect. Providers were asked to accept, modify or reject AI suggestions and to provide a level of confidence in their decisions.
The team found that, using local explanations, providers achieved greater accuracy, at 92.8%; in comparison, global explanations yielded an accuracy of 85.3%. However, when the AI suggestions were incorrect, provider performance significantly declined, regardless of explanation type.
Providers were more likely to accept a diagnosis when local explanations were available, and they tended to do so more quickly as well. While this can improve workflow efficiency, quickly accepting AI advice also can negatively affect diagnostic accuracy.
“When we rely too much on whatever the computer tells us, that’s a problem, because AI is not always right,” Yi says. “I think as radiologists using AI, we need to be aware of these pitfalls and stay mindful of our diagnostic patterns and training.”
The group suggests that their results related to the impact of explanation type should be considered by AI developers in the future.
“I really think collaboration between industry and health care researchers is key,” Yi says. “I hope this paper starts a dialog and fruitful future research collaborations.”
The study abstract is available here.
In addition to her background in journalism, Hannah also has patient-facing experience in clinical settings, having spent more than 12 years working as a registered rad tech. She began covering the medical imaging industry for Innovate Healthcare in 2021.