Natural language processing tool enables recommendation analysis

According to Benjamin Franklin, time is money and an ounce of prevention is worth a pound of cure. Diagnostic imaging practices in the United States are acutely aware of how little time can be wasted in the course of day-to-day operations to efficiently manage study volumes and remain profitable. They’re also cognizant that navigating around roadblocks is more effective than coming up against them and searching for detours.

Analyzing the patterns of a radiology department’s recommended imaging tests, and the time frame in which they’re conducted, can help in assessing the appropriateness of referral practices and radiology self-referral policies, according to Praya Dang, MBBS.

Dang and colleagues from Massachusetts General Hospital (MGH) in Boston utilized the Leximer natural language processing engine (NLPE), developed at the facility, to categorize radiology reports with recommendations based on recommended imaging and time frame. In conjunction with on-line analytical processing, the reviewed 4.2 million radiology reports from 1994-2005 that were in the MGH database.

“We assessed the accuracy of the NLPE for recommended imaging tests and time frame,” Dang said in a presentation at the 2007 Radiological Society of North America scientific assembly. “In addition, we also determined trends for different time frames for recommendations in an 11-year reports database.”

From the radiology reports database, 120 reports with and without recommendations for further action were selected and randomized by the researchers. These reports were classified by two radiologists, as the standard of reference, and the NLPE independently on the basis of presence or absence of recommendations, she said.

Dang reported that the reports with recommendations were further categorized for the imaging modality and time frame recommended. Sensitivity, specificity, accuracy, positive predictive values (PPV) and negative predictive values (NPV) of the classification performed by the NLPE was determined by the researchers using an online statistical calculator (VasserStats).

The NLPE also was used to determine the recommended imaging and time frames for further evaluation or follow up in all the reports, she said.

For recommended modalities, the NLPE had a sensitivity of 100 percent, a specificity of 84.2 percent, an accuracy of 95 percent, a PPV of 93.2 percent, and a NPV of 100 percent.  For time frames, it had a sensitivity of 100 percent, a specificity of 86.48 percent, an accuracy of 95.8 percent, a PPV of 94.3 percent, and a NPV of 100 percent, she reported.

Dang noted that in 21.1 percent of the cases, the radiologists did not specify the recommended modality, and that in 88.7 percent of the cases, they did not specify a time frame for the recommended procedure. In addition, she said that specific time frames were not stated for most recommended CT and MR examinations.

“Unfortunately, most radiology reports with recommendations are non-specific for both modality and time frame,” she noted.

The study showed that the use of an NLPE can help a diagnostic imaging group analyze its recommendations and the time frame in which they’re conducted, allowing it to focus its resources on coaching those referrers who are not following the recommendations in a timely manner. In addition, it can help ensure that interpreting physicians in a group are specific in their recommendations for the modality and time frame in which additional imaging tests should be conducted.