AI 'candidate' fails to pass mock radiology boards
Despite the infamous 2016 prophecy of deep learning expert Geoffrey Hinton, artificial intelligence has not yet replaced radiologists. And according to new data, it appears as though that prediction is still a long way off, as an AI “candidate” recently failed its mock radiology boards.
The candidate's results were published this week in The BMJ and compared alongside 26 radiologists who had recently passed the rapid radiographic reporting component of the Fellowship of the Royal College of Radiologists (FRCR) examination. Out of ten mock exams, the AI candidate passed two, achieving an overall accuracy of 79.5%, suggesting that the candidate is not quite “ready to graduate.”
“Radiologists in the UK are required to pass the Fellowship of the Royal College of Radiologists (FRCR) examination before their completion of training, which allows them to practice independently as radiology consultants. For artificial intelligence to replace radiologists, ensuring that it too can pass the same examination would seem prudent,” corresponding author Susan Cheng Shelmerdine, a consultant pediatric radiologist at the Great Ormond Street Hospital for Children in London, and colleagues suggested.
The rapid reporting section of the FRCR exam requires candidates to interpret 30 radiographs in 35 minutes, achieving 90% accuracy (27 out of 30) in order to pass. The authors explained that this part of the exam is considered a “stress test” for candidates and is an informative measure of candidates’ speed and accuracy—two things often attached to AI descriptions—in fast-paced, stressful situations.
The AI candidate recorded a sensitivity and specificity of 83.6% and 75.2%, respectively, and appeared to struggle with musculoskeletal interpretations the most. The human radiologists outperformed the AI candidate across most measures, although they did not pass every mock examination either.
The authors explained that the results for both the AI candidate and the radiologists could be due to the level of exam difficulty, noting that the set of exams used for the analysis were particularly challenging.
Experts involved in the study were not discouraged by the AI candidate’s performance and suggested that had the AI been trained on a more diverse set of images, it likely would have performed better.
“Although lowly ranked for diagnostic accuracy rate overall (rank 26), the artificial intelligence came close to radiologist level performance when we consider the cases it could interpret,” they indicated, adding that with more training, the future potential of AI as a support tool in real-world scenarios “remains high.”
To learn more, click here.
In addition to her background in journalism, Hannah also has patient-facing experience in clinical settings, having spent more than 12 years working as a registered rad tech. She joined Innovate Healthcare in 2021 and has since put her unique expertise to use in her editorial role with Health Imaging.