Experts cite Google-led breast cancer screening study in call for more AI research transparency
A study claiming artificial intelligence can beat trained radiologists at detecting breast cancer has caused scientists from around the globe to demand more transparency and reproducibility in AI-based research.
Experts at Johns Hopkins, Harvard School of Public Health, MIT, Princess Margaret Cancer Center, and others specifically want journals to hold investigators to higher standards. They implored their colleagues to share codes, models and test settings in peer-reviewed publications.
The groups shared their thoughts in an article published Wednesday in Nature.
"Scientific progress depends on the ability of researchers to scrutinize the results of a study and reproduce the main finding to learn from," Benjamin Haibe-Kains, PhD, a senior scientist at Princess Margaret in Toronto, said in a statement. "But in computational research, it's not yet a widespread criterion for the details of an AI study to be fully accessible. This is detrimental to our progress."
In their piece, the authors pointed out an example of poor transparency that particularly boiled their blood: a Google Health-led study released earlier this year that claimed an AI system—DeepMind—could beat out radiologists at spotting breast cancer on mammograms.
The problem, Haibe-Kains and colleagues noted, is that the study didn’t describe its methods and omitted the tool’s coding and models. This stifled others from understanding how the algorithm worked and if it could be applied at other healthcare institutions, the authors noted.
Health Imaging covered this study back in January, which also included an invited commentary published in Nature warning clinicians of its many limitations.
“The real world is more complicated and potentially more diverse than the type of controlled research environment reported in this study,” Etta D. Pisano, MD, of Harvard Medical School, explained.
In the piece published on Oct. 14, the group offers up frameworks and platforms for researchers to safely share the data, computer code and predictive models used for AI-based investigations. Doing so may help translate valuable findings to improved clinical outcomes.
"We have high hopes for the utility of AI for our cancer patients," Haibe-Kains explained. "Sharing and building upon our discoveries--that's real scientific impact."
Matt joined Chicago’s TriMed team in 2018 covering all areas of health imaging after two years reporting on the hospital field. He holds a bachelor’s in English from UIC, and enjoys a good cup of coffee and an interesting documentary.