Disaster Recovery Earns a Place in the Capital Budget

Disaster recovery sometimes sounds like something only businesses in hurricane-ravaged areas need worry about. However, as you probably know, having a strategy in place is required by HIPAA and it makes sense to have a plan for recovering your patient information data after establishing the infrastructure necessary to collect and store it.

“The state of disaster recovery in the healthcare provider community is a bit of a disaster,” says Barry Runyon, research director of healthcare for The Gartner Group, a consulting firm based in Stamford, Conn. A lot of plans are unrealistic, untestable and naive, he says. “We’re collectively keeping our fingers crossed” that a disaster will not occur. Many CEOs say to Runyon that they haven’t had a disaster so why should they anticipate one? “That’s a hard attitude to overcome,” he admits.


Offsite replicating

But many facilities have overcome that attitude and made disaster recovery a priority. For example, Midwestern Regional Medical Center in Zion, Ill., is using a hosted information management solution for recovery and backup. The 250-bed facility is replicating PACS data to a Kodak-hosted storage facility. When going through their PACS selection process, they also were in the process of moving data. So, they decided to let Kodak host the information until they could create a better data center, says Lawrence White, MCSE, CCNA, CCSA, RIS-PACS information systems manager.

White and the rest of the evaluation team — which included the chief operating officer, vice president of finance, and imaging director — looked at vendors’ support capabilities, years spent developing PACS, and the steps being taken to comply with IHE (Integrating the Healthcare Enterprise) standards. They eventually decided to take advantage of Kodak’s storage expertise, the ability to store data in another region, and the fact that Kodak stores on spinning media. White hasn’t had the opportunity to test how fast he can retrieve previous studies but he has prefetching rules set up so that patient information is retrieved the night before a scheduled study.

White is replicating data for offsite storage at night because he initially only had a T1 connection. He now has a T4 connection but still sends everything at night — about 1 or 2 terabytes a day — and uses the T4 connection during the day to support remote users. That has been the best method because he does not have a third shift of IT staff.

In the event of a disaster, White has an exact copy of everything he would need to rebuild Midwestern’s server. “If the server should go down, we have a cold spare we could rely on.” He would then manually switch everything over and bring up the cold spare, rebuilding from Kodak’s enterprise information management module. Later, he can switch back to the original server. White says that last year’s hurricanes reassured him that this process was the right choice.


Redundancy and more redundancy

Universal Health Network, a three-hospital system based in Toronto, was seeking a vendor that offered storage redundancy and disaster recovery planning for its medical imaging. The organization stores about 150 gigabytes of data a day and performs about 600,000 imaging studies a year. They chose a Network Appliance solution that “allows us to have a disaster at one facility and still maintain data and continued functioning without users really noticing,” says John Adziovsky, RIS-PACS manager.

Adziovsky recommends looking into systems that offer higher data availability and redundancy. Plus, consider the type of organization. “Where does it make sense to put the money for the best patient benefit?” he asks. For his organization, with three emergency departments and high caseloads, they couldn’t afford not to spend more money to put in a redundant system.

Adziovsky’s focus was higher availability of data because of the type of data he’s working with. “It’s critical data that needs to be up all the time.”

He has an internal administrator but cites the excellent service he receives from Network Appliance. “They are constantly monitoring the system and flagging the smallest hiccup.”


Nearby storage, tested system

Phil Hartman, information systems manager at Jefferson City Medical Group in Missouri, reviewed three potential disaster recovery solutions last year when the organization was implementing an electronic records systems. He wanted to upgrade from sending traditional backups to tape. One option, the least expensive, was warehousing data in South Carolina. The second option was warehousing data at a hardened facility in St. Louis. Finally, Hartman could warehouse data at an old underground limestone mine in Columbia, S.C., about 30 minutes away.

Hartman went with the third option because it was closer, it would allow him to stay in a Windows environment, and they had experience with NSI Software’s DoubleTake solution. A major software upgrade last year gave him the opportunity to test the system. He pulled the redundant hardware from the storage site. His organization had decided to go with the exact same hardware and patches, even down to drive speed so they could count on similar performance history. Hartman used the redundant hardware and pretended the server went down. He hooked it up and let the DoubleTake software synchronize with the downed server. “That got our confidence level up,” he says. “We can say we’ve tried [our solution out] in a test environment.”  

Hartman says he could have gone with the disaster recovery solution offered by his EMR vendor. However, that based the entire recovery plan on an operating internet. “We could only get to our data through the internet,” he points out, something the practice was not willing to risk. With the current plan, if a repair or replacement can’t happen within four hours, Hartman hits the road to retrieve the redundant system.


More action in planning

Runyon predicts “a persistent upward curve in the disaster recovery planning industry over the next 2 to 5 years. More healthcare providers will have more actionable disaster recovery plans and capabilities than they have now.” He also thinks that disaster recovery is moving to a capital budget item rather than an operational cost, and graduating from backup and recovery. “Now we’re talking about ensuring availability and integrity of patient healthcare information.”

The problem is that the average healthcare provider becomes overwhelmed, he says, and then gets into a worst-case paralysis situation. Most facilities have 10 or 12 mission-critical systems and another 50 to 60 department systems. “You have to decide what takes priority. Spend more on recovering them. You can do without some systems for a week or even a month.”

Hartman recommends going on site visits to see how your hardware and data would be stored. That gave him the opportunity to see how others were storing their data and realize what protections he and his team were comfortable with.

Realize that storage and disaster planning is a “financial investment,” Hartman says. “You’ve got to have the will to spend that money” and cover recurring fees, such as licensures and data lines.