Addressing 'model drift' to recover AI performance before it leads to report errors
Regularly assessing and updating machine learning models can prevent harmful performance deterioration over time.
This was the case recently for an institution that earlier developed and trained an algorithm to scan radiology reports for follow-up recommendations. It was noted that the accuracy of the model had started to decline over time—a phenomenon that was referred to as “model drift” recently in a paper published in the Journal of the American College of Radiology [1].
“Model drift reflects the observation that no model lasts forever. It consists of data drift or concept drift,” corresponding author Ronilda Lacson, from the Department of Radiology at Brigham and Women’s Hospital in Boston, and co-authors explained. “Although regularly assessing and updating these models is necessary to ensure accurate performance, there is no standard approach to addressing model drift.”
Lacson and colleagues sought to address their original model’s declining precision and recall by retraining it using data augmented with more recent data. To compare its performance, the researchers also developed new support vector machine (SVM) and random forest (RF) models, which were trained with both new and augmented data.
Solutions Are Not Universal
Upon comparison, the experts noted that the RF model trained with augmented data yielded significantly better recall than their baseline model (0.80 versus 0.66) and similar precision (0.90 versus 0.86).
“This highlights the need to comprehensively and methodically try different approaches,” they wrote. “This includes using the same algorithm by recalibrating the same model as well as refitting the model with new data.”
The authors added that retraining an entirely new model is more labor intensive but was worthwhile in their case. However, depending on the situation, updating a baseline model with new data might also suffice—it really just depends on the unique model and situation, the authors suggested before concluding:
Although recalibration or refitting models with new data may be sufficient in some cases, training and validating new machine learning algorithms may be necessary in other cases to address model drift and ensure predictive model performance.”
The study abstract can be viewed here.
In addition to her background in journalism, Hannah also has patient-facing experience in clinical settings, having spent more than 12 years working as a registered rad tech. She joined Innovate Healthcare in 2021 and has since put her unique expertise to use in her editorial role with Health Imaging.