The latest iteration of ChatGPT performed well when tasked with identifying incidental findings on CT scans, leaving researchers from Canada to wonder if the popular AI chatbot could speed up radiology workflows. The results are published in the American Journal of Roentgenology. [1]
ChatGPT-4 was trained to read CT reports using a process called “single-shot learning” wherein the AI is trained using two images for comparison repeatedly, until it is able to identify distinct differences that, in this case, may be relevant to radiologists and patients.
“Automatic identification of incidental findings in radiology reports could improve patient care by highlighting the findings to referring clinicians, automating management, or facilitating population health initiatives,” the research team led by Rajesh Bhayana, MD, from Toronto General Hospital wrote.
For the study, the researchers randomly selected and de-identified 1,000 radiology reports consisting of CT scans of the abdomen, all taken from the hospital record system. A seasoned radiologist with years of post-training experience, along with a physician with four years of practical expertise, independently examined the reports to establish the reference standard.
The duo emphasized key elements important to clinicians, specifically any potential new malignancy on exams stemming from an adrenal nodule measuring less than 1 cm, a pancreatic cystic lesion or signs of vascular calcification.
Later, a second radiologist with 10 years of clinical experience resolved any discrepancies before ChatGPT-4 was tasked with analyzing the images. F1 scores were utilized to measure accuracy, with a 1 indicating the large language model was able to identify potential concerns during its scan for incidental findings with expert precision.
In its analysis, GPT-4 performed nearly perfectly, achieving F1 scores of 1 for identification of adrenal nodules from CT reports, followed by 0.99 for vascular calcification and 0.91 for pancreatic lesions, with radiologists confirming the results.
The authors noted the performance was similar to a trained radiologist, and the ability of ChatGPT-4 to perform this initial sweep of images could reduce errors through additional identification, in addition to saving radiology departments time by implementing the AI into workflows after CT scans.
“Since incidental findings are commonly mismanaged, automatic identification in reports could improve management by increasing visibility or partially automating workup,” the researchers wrote.
“As large language models are increasingly deployed in EMRs, their flexibility will provide new opportunities to improve care,” they concluded.
The full study can be read at the link below.
Chad is an award-winning writer and editor with over 15 years of experience working in media. He has a decade-long professional background in healthcare, working as a writer and in public relations.