Shirking Disaster

The traditional approach to disaster recovery, where system architecture is configured to store a copy of imaging data on tape in an iron mountain, may have reached the tipping point - requiring considerable design modifications, propelled by exponential increases in data that must be maintained indefinitely. When an IT professional considers the quantity of information - including not only imaging data, but corporate email, electronic health records, and business continuity information for the entire healthcare enterprise - the magnitude of the challenge attains monumental significance.

Barry Runyon, research director for healthcare in the Gartner Group of Stamford, Conn., explains that with approximately 50 percent of all data now online, the sheer quantitative crush of that data necessitates careful consideration of a disaster recovery strategy. With industry estimates indicating that about 20 percent of health institutions have deployed picture archive and communications systems (PACS) to date, these issues will intensify over the next five years as PACS become an expected portion of the clinical suite and assume "standard of care" status.

Several evolving trends will force the issue of information lifecycle management activities where disaster recovery is only one component, and Runyon offers suggestions about how to meet the challenges looming just around the corner. The first step involves careful planning and assessment to prioritize the specifics of which data are critical to the enterprise, to determine what data must be recovered in what period of time, precisely what needs to be retained, on what media, and for how many years.

Because clinical data, including medical images and other components of the electronic medical records such as laboratory and pharmacy data, and corporate email communication between clinicians and patients are required from a legal perspective to be maintained for many years, the first trend involves integration of all data into a single stream. In other words, Runyon suggests that a PACS should not be isolated within its own disaster recovery plan.  

Another concern arises in the eventuality of a disaster occurrence, the institution would not be able to wait several days for data to be restored - unless the entire healthcare facility were destroyed and could no longer function in any manner. Even then, many of the records maintained by the institution would be required by other area healthcare entities to properly manage patients with chronic and long-term health issues.

"Many large hospitals have established alternate data sites, to back up one another," says Runyon. For example, the cancer center across town will back up the primary IT data center and visa versa. This trend is designed to address a series of issues that include back up and recovery activities as well as off-site storage and archiving with a single architecture. "If it is a big enough enterprise, with a large IT budget and a high level of expertise, they will be doing their own disk-to-disk to tape architecture, which could be disk to near line storage or disk-to-a-remote mirroring storage area network [SAN]."

Gartner storage analysts expect that the use of tape archive copies will remain prevalent, despite the inherent costs and inefficiencies in retrieval of specific studies, and the problems with degradation of tape over the long-term, until the costs of other media storage options decline.


In the field

Scott Stewart, manager of information services at Rochelle Community Hospital in Rochelle, Ill., has designed a system for their small community, critical access hospital by employing Kodak Carestream Information Management Solution from the Kodak Health Group.

With one SAN in the hospital building and another at a clinic about 11/2 miles away, and fiber connectivity between the two locations, the system has designed to provide continual backup.

"We are keeping the two SANs in sync with one another," says Stewart. As the data are written, they are copied to the local PACS server, and then written to the SAN in one location and sent almost instantly to the other SAN located in an old bank building (where the server sets next to a circa 1920 bank vault). The two SANs are synchronized within about 10 seconds. "Within a minute of running the imaging study, it's stored in three locations, and then sometime during the day it's written to tape as the final backup."

This small IT staff of 2.8 FTEs wanted to insure ease of maintenance, and they are in the process of adding all of their other data to the system. Stewart describes a very positive experience with Kodak providing training for their staff and support as they have continued to evolve their system.

"We decided to partition the SAN so that the vast majority of the data was sent to the PACS, but we're also putting together a LUN [logical unit number] where we can put Windows servers and AS400 data and anything else where we can map a drive letter to receive it," explains Stewart.  From there, data go through the rest of the Kodak Carestream IMS to enter the two SANs and long-term tape archive.

Mark Middleton, systems director for data centers and technical infrastructure at Christus Health in Dallas, describes their use of a highly available large ATM (asynchronous transfer mode) network in the state of Texas as the heart of their ability to implement an enterprise storage solution PACS. He describes a strategic partnership with EMC to develop their comprehensive storage and disaster recovery approach.

With 40 hospitals in their system, they established communication hubs with light data processing coupled with data being targeted to their production enterprise data center in San Antonio, and a disaster recovery facility in Houston. Careful up-front planning based on goals they set through visioning processes led to this enterprise approach incorporating the needs of image management required by PACS, and includes document imaging enterprise-wide.

They decided to maintain six months of frequently accessed imaging data on magnetic disk in one of 10 remote data centers stored on an EMC Clariion CX700, to provide LAN-type speeds, explains Middleton. Besides the data being written to the local magnetic disks, they also write to the permanent archive, EMC Centera, in the San Antonio and Houston data centers.  

"We did a seven year needs analysis that drove the launching of Centera over tape," Middleton says. "Today, we have about 40 Terabytes in place, and our seven year estimate is that we'll be at about one-half Petabyte at the end of seven years."

Mark Jennings, CIO of Northeast Georgia Health System in Gainesville, selected SunGard Availability Systems (Wayne, Pa.) to provide their long-term image and data retrieval needs. This two-hospital system near Atlanta just started an open-heart surgery program, and they are considered the tertiary care provider for the northeast portion of Georgia as well as admitting some patients from North and South Carolina.

In their initial planning, the team at Northeast Georgia decided to set aside the traditional approach of disaster recovery, because they recognized that they needed to have continual access to critical data. "We came to the realization that with terabytes of data, it would take quite a time to accomplish full recovery of all data and applications," says Jennings.

With technology enabled by the telecom industry, they accomplish data replication - which negates the need for data recovery.

"Our SAN not only has PACS data, it already has our document imaging, our electronic medical records and several other systems on it," Jennings says. "We're putting the top six to eight critical systems for the operation of our institutions on our SAN."

Clinical Image Management Systems in San Antonio provides the CIMS Offsite Disaster Recovery System to a variety of non-traditional imaging settings as well as rural hospitals via T1 connections provided by TORCH (Texas Organization of Rural & Community Hospitals). Their clients include cruise ships, oil platforms, and medical examiner forensic settings as well as the remote clinics and rural institutions.

Helen T. Brame, RT (R) (M) director of radiology for the Schleicher County Medical Center in El Dorado, Texas, explains that they use an Onyx-Rad server (from Viztek) to store images for the first two months. Because their imaging activities are film-based, they use a film digitizer to convert images into digital data, and send them to a radiologist located 100 miles away over the T1 line for their 150 to 170 imaging studies per month.

After normal hospital hours, the server automatically sends all of the images produced that day to the CIMS off-line storage. This system employs online, near line and offline storage, with secure "bunkered" DVD copies of all of their images maintained offline.

Judith Albarelli, MS, RDMS, sonographer with Ultrasound Association and Women's Imaging in Alexandria, Va., describes their use of PowerArchive Workstation and CD/DVD Generator from RADinfo Systems in Dulles, Va., to produce automatically backed-up copies of all of their imaging studies. As a women's imaging center, they perform bone density, mammography, high-risk ultrasound and routine ultrasound exams.

"All of our images go directly into PowerArchive, and at midnight, 60 to 80 cases per day are burned to DVDs." They maintain images online on the server for seven days, but the DVD copies are kept for years.


Conclusion

The importance of retaining secure copies of healthcare information and medical images for recovery purposes cannot be overstated. These vital activities require careful planning by IT professionals to establish systems that function smoothly in the near-term while providing for any eventuality in the future. Because applications and methods for running current programs could change over years, frequent testing of recovery capabilities must become part of the plan.