How AI 'cheating' could impact algorithm reliability
A new study on the implications of AI shortcutting has experts raising concerns about the integration of the technology into medicine.
AI shortcutting occurs when “DL models inform their predictions with patterns in the data that are easy to detect algorithmically but potentially misleading,” study co-author Peter L. Schilling, with the Department of Orthopaedic Surgery at Dartmouth-Hitchcock Medical Center, and colleagues explain in Scientific Reports. Shortcutting exploits subtle and unintended patterns algorithms pick up on in datasets. Unfortunately, many AI models have a tendency to adopt shortcutting tendencies, which can make them vulnerable to utilizing improper data and inappropriately applying it in their predictions without logic or reasoning. This also raises concerns over the type of data these models are trained on.
“The eagerness with which these models learn to shortcut is important to recognize. Equally critical is how difficult it is to remove these effects,” the authors caution. “Understanding the full extent of how shortcutting happens on medical images is critical to researchers, reviewers, and readers alike.”
Researchers at Dartmouth Health recently conducted an experiment to better explain how AI shortcutting could negatively impact model performance in medical imaging scenarios. They used simple ResNet18 convolutional neural networks to train models to do something that isn’t actually possible—to determine whether patients frequently consumed refried beans or beer based on radiographs of their knees alone.
Although the models’ performances were decent, with an AUC of 0.63 for refried beans and 0.73 for beer, there were no data within the images that would have reliably guided them to their predictions. Essentially, the models identified a series of bogus patterns within the images that had nothing to do with carb consumption. When the team removed certain variables, like the manufacturer of an X-ray machine or images that were obtained at different sites, the models would still pick up on other subtle patterns.
In a subanalysis and without any additional training, the team challenged those same models to predict what year the radiographs were taken. Using variables that were not immediately obvious to humans, the models were successfully able to predict the calendar year when the images were obtained, further emphasizing their shortcutting capabilities.
“These examples show the potential and range of what shortcutting can pick up,” the authors write. “Note that just because a model can learn to spot these variables does not mean it necessarily does when trained for another task; this merely establishes potential.”
The group notes that there are risks associated with using these techniques, especially within the realm of medical imaging, highlighting the need for stringent oversight in their development.
“Traditional methods largely limit us to visual features humans are already aware of. Handing this feature engineering over to the algorithm in its entirety poses something of a double-edged sword: CNNs automatically detect features, features we know we wanted as well as the ones we didn’t know we needed, but it also means we get the features we never wanted and shouldn’t have,” they explain.
The teams adds that, at this time, it is difficult to determine if the subtle patterns AI models pick up on when “cheating” are correlations or coincidences. This further emphasizing the need to explainable AI tools, they suggest.
“Deep learning was designed for prediction, not hypothesis testing,” the group says. “This means that discovery through a black-box tool like CNNs demands far greater proof than simply showing a model found correlations in the sea of data within an image.”
The team also provided some suggestions on how the issue with shortcutting could be improved. Those can be found here.
In addition to her background in journalism, Hannah also has patient-facing experience in clinical settings, having spent more than 12 years working as a registered rad tech. She joined Innovate Healthcare in 2021 and has since put her unique expertise to use in her editorial role with Health Imaging.