Disaster Recovery Plans That Work

The term "mission critical" could have been coined as the definition for maintaining access to electronic medical images every moment of every day. Radiology administrators and IT professionals are charged with configuring their networks to accomplish these important goals, using either commercial PACS design or non-traditional methodology. Most would agree that redundancy is the name of the game.


New York Presbyterian, the University Hospital of Columbia and Cornell in New York City

Steven Hermann, MS, RT, CRA, director of imaging, describes their current network as quite robust, designed with redundancy in archiving so that they can push and pull data from archives in two entirely separate physical locations.

Over the next year to 18 months, they are taking advantage of a shift in PACS solutions to implement new configurations and to increase redundant capabilities and bandwidth across their network. This huge healthcare enterprise complex consists of two primary campuses, the Weill-Cornell campus on 68th Street and the Columbia University campus on 168th Street on the upper West side of the city, with a data center on East 38th Street and several other remote sites.

"The system was designed to have two different archives in two different locations for the images, so that we could route from one archive or the other to the various PACS workstations," says Hermann.

Leo Bodden, network design manager for the system, describes their existing network and their plans for advancements.

Currently at least two circuits from Verizon enter each the buildings from different directions. For example, the 38th Street data center has one circuit that enters from 38th Street and the other on 39th Street to reduce the possibility of disruption.

"From 68th Street to 38th Street, there are four 50 megabit ATM [Asynchronous Transfer Mode] PCVs [Permanent Virtual Circuits]," says Bodden. "That connection is completely redundant, meaning it even goes to diverse COs [Central Offices]." Between 168th and 38th, there are four 20 megabit PVCs. Basically, the network is designed to provide redundant interconnectivity between all sites.

Bodden explains that they are in the process of constructing an optical network throughout New York City, and they've been laying dark fiber that will "go light" this month.

"We're building a dual 10 gig ring between those three major sites, and we are adding gigabit connections over to our remotes," says Bodden. "So we'll go from today where we have 50 meg circuits to 10 gig links." Again, complete redundancy insures high availability.

But they're not stopping there. In addition, they are building a microwave ring between their three primary sites.

"If we do lose the 10 gig connections, we'll have OC-3 microwaves, which are 155 megabits [Mbps] throughout our locations," says Bodden. "If we were to lose the dark fiber completely, we'd still be running." The wireless dishes on the building rooftops are designed solely for redundancy purposes, and although they provide a smaller pipe for data transfer, they would enable continuation of vital functionality.

Bodden explains that their wide area network (WAN) was designed to enable all infrastructure components to communicate, but he advises that the local area network (LAN) should be as robust as well. "We built redundancy in the LAN where we have dual paths to all of our equipment."


University of California at Davis Medical Center

Alex Lopez, system architect at UC Davis, turned to the globally-active Ciena Corp. to enable the design and implement its disaster recovery plan. The facility moves stored images and information from the electronic medical record (EMR) system between two data centers. The challenge has always been to manage huge data storage requirements from cardiology and echocardiography procedures. A Ciena 2000 Storage Extension Platform and other applications move and store that data as well as replicate it for disaster recovery purposes.

"The advantage of the multiplexing is that we have fiber channel protocols, mainframe protocols, and Ethernet protocols that we use for storage replication," says Lopez. "We use the Ciena and convert to fiber channel to move it to the other building, so we don't need to have a separate network for each one of them."

Lopez describes the approach as Ciena utilizes SONET (Synchronous Optical Network), a standard for connecting fiber-optic transmission systems, based on the bandwidth of the pipeline used during transport. UCDavis has an OC-48 network that moves data at 2.44 gigabits per second (Gbps).

"It's rather large for us, but we're planning for the future. We only want to do it once and not replace it," explains Lopez. Because the facility performs a large number of echocardiograms, it has built an archive system on DVD. Before implementing this solution, it required 45 minutes to retrieve prior (older than 90 days) exams for use. Now, they retrieve images older than 90 days from disk through the multiplexes from the secondary building which is far away, in a matter of seconds.

The primary benefit Lopez has found using the Ciena product is the consolidation of protocols, meaning they don't have to provide a separate network for each requirement. For example, mainframe applications use ESCON protocols, Ethernet another, and fiber channel is used for open systems like UNIX.

"We would have to buy repeater boxes for each protocol, and the cost is high," says Lopez. "Then I found this Ciena product that was simple to use and achieved what we needed. And it really works!"


Gulf Coast Region of Christus Health in Texas

This healthcare enterprise consisting of three facilities within a 50-mile radius, Christus St. Joseph Hospital (300 beds) in Houston, Christus St. John Hospital (125 beds) in Clearlake and Christus St. Catherine Hospital (60 beds) in Katy has built a robust disaster recovery program. All three facilities have gone filmless using the McKesson Horizon Medical Imaging PACS.

"From the standpoint of how we handle protection from disaster is that even though the facilities are located in different areas, we have located our image servers in a data center in another facility in Houston," says Tom Shirley, market information officer for the Gulf Coast Region of Christus Health. "Our back-ups for those images occur in our enterprise data center in San Antonio."

Evelyn Briggs, director of security and compliance for the network, explains that their first tier databases are replicated at the Houston data center to provide a secondary duplicate to back up their primary storage system in each hospital. Then they have a tape back up at the enterprise data center in San Antonio.

Shirley advises that besides making plans for disaster recovery of the images, institutions should prioritize the applications that would be important to operations, and back those up as well. In other words, an institution would need the images and applications to run the PACS, so all of those components of the system should be included in the disaster recovery plans. The same would be true for the admission system and other clinical applications.


Mercy Medical Center in Canton, Ohio

Bruce Stefancik, radiology PACS manager, describes their disaster recovery approach that utilizes the Philips EasyAccess PACS version 8.2 on its 1-gigabyte fiber optic backbone. They use a StorageTek tape archive that makes two copies that are stored in different sites. They also use a TeraRecon server that functions for back-up storage.

One of the other strategies Stefancik employed was to increase the memory on their CR units, so that they could hold several days of images at the modality, without slowing down the CR scanner.

The Philips EasyAccess PACS is designed with redundancy in every element, so that the failure of any single component does not compromise the overall operations of the system. Using a technology called Load Balancing, the core components can be physically divided and located in two different rooms or floors of the building or even two separate buildings on the network.

In the event of a catastrophic event such as a technical breakdown, or occurrence such as a fire or flood, where one of the central storage sites becomes nonfunctional, the other component would automatically assume the entire load and provide all of the services that were required. This Active High Availability is offered standard with every Philips Enterprise RAID/PACS at no additional charge.


DOSHI Diagnostic Imaging Services in Hicksville, N.Y.

Hemant Maru, who is director of purchasing and operations, describes their use of the Kodak DirectView PACS with Enterprise Information Management (EIM) Services to provide appropriate administration for medical images and information stored on diverse platforms at multiple locations.

"Kodak's solution with VIParchive software offers several important advantages," says Maru. "The software is very scalable and allows us to add new storage technologies or devices at any time. It also enables us to centralize storage." While they have redundancy built into this system, DOSHI is in the process of designing a new off-site redundant center, which they believe will be facilitated by this approach.

EIM Services include installation and integration of the Kodak DirectView Versatile Intelligent Patient Archive (VIParchive) information management software. At DOSHI, this approach is used to manage and store 35,000 imaging studies a month that are acquired in nine multi-modality imaging centers in New York City and Long Island. The flexibility this system permits has proven valuable to sites such as DOSHI.


Conclusion

In this era of "anything can happen ... and it probably will," network managers are charged with maintaining continuity of the system despite surrounding events. Disaster recovery requires careful planning and a bit of ingenuity to insure constant access to medical images and other clinical data. System administrators have risen to the challenge with a variety of approaches, and contingency plans are poised to save the day and turn the IT department professionals into heroes.