Concerns raised over how hospitals can validate radiology AI algorithms

As artificial intelligence (AI) adoption expands in radiology, there is growing concern that AI algorithms needs to undergo quality assurance (QA) reviews just like imaging systems to ensure they are performing as intended and the data they produce is accurate. The American College of Radiology (ACR) has established the Assess-AI Registry and AI-Lab in an effort help with validating and tracking AI QA for FDA-cleared algorithms.

Today, there are more than 340 AI algorithms cleared by the U.S. Food and Drug Administration (FDA), and a large number of these are for medical imaging. 

"Radiologists want to know AI will be safe and effective for their patients. In other words, they want to make sure it's trust worthy," explained Bibb Allen, MD, FACR, chief medical officer of the ACR Data Science Institute and former ACR president. 

According to Allen, there are lingering concerns that medical imaging AI algorithms may have been validated on datasets that were too small, limited by single center evaluations using specific scanners and settings or with limited diversity in the datasets. Allen said his hospital in Alabama has a much different patient population with a very high rate of obesity and metabolic issues as compared to patients from a hospital in Utah or the West Coast. 

"It's not surprising that an algorithm performs well in a test environment," Allen explained.

He said AI vendors will tell you their products work better than average quality benchmarks based on their testing.

"Well that's great, but come and see what will work at our practice in Alabama where we have rampant hypertension and obesity," Allen said. 

He explained there are not standard industry pathways to validate AI to offer assurances that it will work and be trustworthy in all settings. There are variables in patient populations, between males and females, by region, ethnicity and socio-economic levels. 

"We are recommending to the end users that they evaluate the models using their own patient data and by using an enriched dataset," Allen explained. "We want them to put in the hard cases so they can say 'OK, the AI found this one, and this was a case that was hard for me.' Equally important is the real-world monitoring."

He said data drift, changes in imaging equipment used, settings on the scanners and software updates might have an impact on the AI software. So, like the scanners themselves, or even the PACS monitors used to read studies, QA needs to be performed to guarantee the AI is working properly. This is especially important when it is relied on for critical findings, computer-aided detection (CAD), when modifying images for things like bone removal, or performing complex qualification that is used to determine patient treatments or next steps. 

"The AI is always going to work best the day you plug it in, and then there is going to be some performance degradation over time. So keeping track of that and understanding why it breaks is important," Allen said. 

For example, if you have a portable X-ray system for chest X-rays with a built-in AI pneumothorax detector, it is possible that over time it might not work as well as it used to. 

"If you have the data on the AI model's performance in a registry where you track each portable X-ray system, you might recognize that this is not the same X-ray unit as the one you did a software upgrade, and now the model is not performing as well as it used to," Allen said. "The registry can help you figure out what you changed and what you need to do to fix it."

He said AI developers also could use these insights to build better algorithms and understand what technical factors may impact the performance of their AI.

These are the goals of the ACR Assess-AI Registry, Allen explained. 

Like the medical imaging radiation dose index registry and others created by the ACR, he said these are tools to help set baselines and monitor quality over time. And not just for one site or health system, but for all of radiology, because the pooled data will help identify problems and solutions over time. 

ACR offers radiology AI test platform to evaluate algorithms

The ACR Data Institute has created a searchable online catalog with information on all the FDA-cleared AI algorithms for radiology. Allen said it is too difficult to test all the AI models that are out there, so this can help narrow the list. 

The ACR Data Science Institute also has a website called AI-Lab, which allows people test an algorithm on premises or in a secure cloud using their own dataset. Allen said users also can compare one algorithm to another to test their performance. 

"Because the models can be brittle and they may not work as well outside of where they were trained, you will want to be able to do an evaluation," Allen said. "AI-Lab allows people to query, retrieve and create a test set and then actually run a model and test the performance. You also can compare one algorithm to another."