Researchers cite safety concerns after uncovering 'harmful behavior' of fracture-detecting AI model

Researchers are cautioning the overly optimistic AI enthusiasts after a recent algorithmic audit revealed the potentially “harmful behavior” of a validated model intended to detect femoral fractures. 

A study in the Lancet Digital Health reports that a previously validated, high performing AI model committed troublesome errors when confronted with atypical anatomy while seeking out subtle proximal femur fractures. Researchers noted that despite the model’s exceptional performance on external validation, its preclinical performance revealed barriers that would inhibit the algorithm’s ability to be safely deployed in clinical practice. Experts involved in the study acknowledged that this is a common obstacle when transitioning artificial intelligence systems into every day, real world clinical practice. 

“Historically, computer-aided diagnosis systems have often performed unexpectedly poorly in the clinical setting despite promising preclinical evaluations, a concept known as the implementation gap,” corresponding author Lauren Oakden-Rayner, MBBS, with the Australian Institute for Machine Learning in Adelaide, Australia, and co-authors explained. “Few preclinical artificial intelligence research studies have addressed these concerns; for example, external validation—an assessment of the ability of a model to generalize to new environments—has only been done in around a third of studies.” 

Algorithmic auditing has been presented as a possible solution to help overcome the problem of poorly performing computer-aided systems in clinical settings. Auditing can identify and mitigate any issues that would cause the algorithm to maintain diagnostic performance. That’s exactly what researchers did with their preclinical evaluation of a previously validated deep learning model developed to detect proximal femur fractures based off frontal x-ray films in emergency settings.