AI developed during high-profile competition rivals radiologists’ lung cancer predictions
Artificial intelligence algorithms developed during high-profile competitions rival radiologists’ ability to spot lung cancer on screening exams, according to data published Wednesday.
Back in 2017, participants in the Data Science Bowl were tasked with developing software that could pinpoint cancerous lung lesions on low-dose chest CT scans. Winning algorithms were announced, but organizers never tested their real-world performance nor if they stood up to independent imaging data.
Experts took three of the top performing tools and pitted them against 11 radiologists. Each produced a score from 0-1 indicating the likelihood of developing lung cancer within a year. Two of the three rivaled the predictions made by expert rads.
The findings present several opportunities to optimize LDCT interpretations, even though the models cannot locate cancers nor explain why a score was given, researchers wrote in Radiology: Artificial Intelligence.
“Therefore, this score can, at present, only be used as a sign that a radiologist needs to carefully check the CT scan for abnormalities,” Colin Jacobs, PhD, with Radboud University Medical Center’s Department of Radiology in the Netherlands, and co-authors noted. “Alternatively, these algorithms could be used to triage normal scans and only send possibly abnormal scans for radiologist review,” the group added.
The team retrospectively assessed the top deep learning tools using 300 LDCT scans (150 from the competition, 150 from an independent set). Both datasets included 50 images with cancer and 100 without.
Algorithm 1 (grt123) achieved an area under the receiver operating characteristic curve score of 0.88, while algorithm 2 (JWDH) notched a 0.90 and algorithm 3 (Aidence) a 0.92.
By comparison, radiologists’ average AUC proved to be 0.92. This significantly outperformed one deep learning tool but wasn’t much different than JWDH or Aidence, the authors noted.
Jacobs and colleagues explained that such tools may be best used to triage normal scans and send abnormal images for human review. This practice may substantially reduce the costs of screening but requires further investigation.
“If future validation studies show that this approach is feasible, policy changes will be needed because at present, every screening CT scan must be categorized according to Lung-RADS by a board-certified radiologist in the United States to qualify for reimbursement,” the authors noted.
Read the full study here.
Matt joined Chicago’s TriMed team in 2018 covering all areas of health imaging after two years reporting on the hospital field. He holds a bachelor’s in English from UIC, and enjoys a good cup of coffee and an interesting documentary.