ACR partners to create AI foundational model assessment website
In this video interview, Christoph Wald, MD, PhD, MBA, FACR, vice chair of the American College of Radiology Board of Chancellors and chair of the ACR Commission on Informatics, explains that ACR’s partnership with several academic institutions to create the Health AI Challenge will help provide a better understanding of how foundational AI models work.
The challenge website allows numerous foundational AI models to be tested on the same image datasets to see how they perform, also giving users opportunities to rate the quality of the AI analysis.
"It's a very interesting new concept when you think about AI foundation models. These are coming from nontraditional sources in many instances, and if you were to ask the developer, 'How does it work?' there's probably no answer to this. So, we created a computational space where we stage a radiology problem. The most simple one is for chest X-ray, so we stage a bunch of chest X-rays and have the reports on the original studies. We then let a number of foundation AI models that claim to be able to interpret chest X-rays run on that dataset," Wald explained.
He said the website then presents the blinded outputs that do not tell the user if they came from a foundational model or from a radiologist. ACR and other partner organizations then invite all radiologists to join the challenge page so they can look at that problem and score the results. He said this includes rating the results with comments describing how they look, whether that's good, bad, first year resident attending-level, terrible or not actionable.
"We're crowdsourcing how well an AI model is working. So, if we can't explain how these things work, at the very least we can say they do work, and what's happening there," Wald explained.
He said this gamification of looking at AI will use the feedback to determine how well foundational models work on a particular problem.
Read more details in the article ACR, top health systems form collaborative to help radiologists assess AI solutions.
Dave Fornell has covered healthcare for more than 17 years, with a focus in cardiology and radiology. Fornell is a 5-time winner of a Jesse H. Neal Award, the most prestigious editorial honors in the field of specialized journalism. The wins included best technical content, best use of social media and best COVID-19 coverage. Fornell was also a three-time Neal finalist for best range of work by a single author. He produces more than 100 editorial videos each year, most of them interviews with key opinion leaders in medicine. He also writes technical articles, covers key trends, conducts video hospital site visits, and is very involved with social media. E-mail: [email protected]