Business Continuity: High Availability for Clinical & Business Applications

In the last few decades healthcare organizations have deployed information technology (IT) to meet clinical and business needs across the enterprise—in hospitals, imaging centers and physician offices, large and small. A growing reliance on IT infrastructure has increased sharply in the last decade, as more facilities embrace PACS (picture archiving and communications systems), CPOE (computerized physician order entry) and EHR (electronic health record) systems.

And with the adoption of sophisticated IT applications, healthcare organizations have been enabled to improve patient care and safety, trim costs and share information among authorized caregivers. These benefits, however, are not without drawbacks. Increased utilization of IT solutions translates into increased dependence, and thus risk, in the event of an outage or data loss.

What are the risks to a healthcare enterprise when an outage occurs or a storm wipes out critical data in the data center? Many facilities are gradually eliminating paper, so simply returning to old processes during a system outage is not as easy as it sounds. In fact, returning to paper processes can be costly and may even be impossible. “We’ve been migrating to full automation and will reach the point where our staff won’t be as familiar with manual procedures,” explains Chris Panagiotopoulos, director of technology for LifeBridge Health in Baltimore.

Other options can be equally distasteful. The enterprise can accept the risk of an outage, which may open the door to unpleasant and expensive economic and legal ramifications.

In today’s healthcare environment, critical clinical data must be recovered in a matter of minutes. Non-mission critical systems like patient registration or supply chain management may allow for a slightly longer recovery time, but recovery must be timely. That’s because an inventory outage could occur while a time-consuming manual data recovery process is being conducted.

A solid business continuity plan provides the healthcare organization with the redundancy and infrastructure necessary to recover data if a primary system or data center is compromised or destroyed. A disaster recovery contingency plan also is needed to comply with the Healthcare Insurance Portability and Accountability Act (HIPAA) requirements to respond to system emergencies.

When the physician owners of Austin Radiological Association in Austin, Texas, pondered the implications of operating a highly IT-dependent enterprise, they opted for the business continuity route, says CIO Todd Thomas. “We made a strategic decision to implement a business continuity plan,” continues Thomas. The CIO likens business continuity to an insurance plan; the buyer hopes he’ll never need it, but appreciates its value if it is used.

Listen to an interview with Todd Thomas of the Austin Radiological Association.

Furthermore, the benefits of business continuity transcend its insurance value. Many facilities that have turned to industry leaders like EMC to help deploy business continuity solutions are reporting benefits—including better utilization of IT assets, increased availability of systems and reduced costs.

Achieving business continuity is a complex, but increasingly necessary, undertaking. It is an ongoing strategy that includes education, assessment, planning, implementing and testing. Healthcare organizations need to consider server and storage platforms, recovery plans and tools, network management and support services.

Building the business continuity plan

The transition to business continuity begins with building the case for business continuity. After key decision-makers understand the risks and consequences of outages and make the strategic decision to implement business continuity, the next step is to plan and engineer the system. Early in the plan the hospital needs to develop an understanding of user requirements and how quickly data must be recovered. Healthcare organizations need to classify data according to the clinical and business value and create performance level requirements and protocols to manage data from its creation. Next, healthcare organizations need to determine the cost of data loss if systems go down, including how much data loss is “affordable” and what is the longest tolerable outage.

Joe Wagner, CTO of El Camino Hospital in Mountain View, Calif., observes, “The business continuity plan is centered on and built around clinical needs and workflow.” El Camino Hospital began with patient care, determining clinical needs with data implications and worked backwards from that point. “We looked at more than 20 independent business units in the hospital and developed an understanding of the ramifications of a failure,” Wagner says. In the radiology department that assessment led to a 99.999 percent uptime for PACS, which meets their demanding recovery objectives.

El Camino Hospital decided to transition from a storage environment that combined direct-attached storage to server, towers and CD racks to a consolidated EMC storage infrastructure. Data replication represented a difficult task in the previous data center as each storage island required individual backup and then data was mirrored or arrayed.

The cornerstone of the new plan is EMC CLARiiON CX series-based storage area network (SAN). EMC SnapView provides synchronous data backup to advanced technology attachment (ATA) drives. This model allows the healthcare organization to create multiple tape backups for offsite storage to meet business continuity needs. El Camino Hospital, however, aims to deploy a more robust system in the future; the hospital plans to build an offsite data center to house replicated images and data. EMC MirrorView software will immediately replicate any disk at the remote site.

El Camino Hospital’s consolidated storage infrastructure allows IT staff to support and maintain larger volumes of applications and data without a corresponding increase in staff, saving the hospital an estimated $150,000 annually. In addition, the hospital has saved 20 to 30 percent in total square footage in its data center because its solutions consume less space than previous storage servers, towers and racks.

This state-of-the-art business continuity plan is in contrast to tape or DVD-based plans. Take, for example, the prior environment at LifeBridge Health. “Two and a half years ago, most of the systems were attached to Symmetrix for backup to tape,” recalls Panagiotopoulos. “Recovery was long and arduous.”

LifeBridge Health decided it needed to simplify storage management and contain costs. The answer was an upgrade to a 25-terabyte EMC Symmetrix storage platform. The new system not only delivers improved performance and additional capacity but also facilitates the healthcare network’s business continuity plan.

A remote site houses a second Symmetrix, and EMC SRDF/Synchronous remote replication software continuously mirrors and protects applications and data, helping the site meet HIPAA contingency plan requirements. The main data center and primary Symmetrix are connected to the remote site via a broadband connection, with a second vendor providing a back-up line for data transmission.

One of the key elements of LifeBridge Health’s business continuity strategy is ongoing testing. “We aim to stay sharp, and the team has surpassed our original expectations,” notes Panagiotopoulos. Phase one of the testing plan was built around declared simultaneous disasters. The IT department times the recovery process, consisting of systems at the remote site, splitting the SRDF link, facilitating full production and sending test patient information to ensure interoperability and data consistency. Once the department became familiar with the process, Panagiotopoulos held unannounced disaster tests. “Our expectation was that the recovery process would take four to six hours, but we demonstrated that we can recover systems within one to two hours,” he says. The final challenge was the unplanned disaster on days when primary team members are absent. After a few ‘disasters’ the performance of the second and third level support teams has improved tremendously, says Panagiotopoulos.

Listen to an interview with Chris Panagiotopoulos of LifeBridge Health.

Defining business continuity

Four years ago, University of Chicago Hospitals and Health Systems (UCHHS) embraced business continuity as an enterprise goal, defining business continuity as traditional disaster recovery working along with downtime procedures. “Our main focus is to make sure data are available in the event of a disaster or application failure. We needed faster, very reliable data recovery,” explains Todd Hollowell, executive director, information technology. University of Chicago Hospitals also aimed to phase out the best-of-breed data storage approach, which carried higher costs and more rigorous support requirements. Today, the academic medical center deploys a multi-pronged approach, leveraging an EMC intelligent information infrastructure.

As UCHHS began planning its business continuity solution, it classified applications into groupings with similar storage and recovery needs. Symmetrix DMX hosts data generated by high-speed transactional applications that require the highest recovery and continuous accessibility. Applications like the clinical information system and the Hospitals’ financial system fit into this top tier.

At the next tier, an EMC CLARiiON CX platform holds applications like echocardiography, GI and lab systems that require slightly less demanding availability and performance requirements. For the third tier, EMC Centera provides an online solution for fixed content like email, archived echo images and web content. This active archive enables users to leverage the storage environment with little perceptible difference in performance. This multi-tiered model offers a cost-effective way to manage data while maintaining high utilization, summarizes Hollowell.

UCHHS deploys EMC TimeFinder, Legato backup solutions and Symmetrix Remote Data Facility/Asynchronous (SRDF/A) recovery software to complete the site’s robust business continuity plan. TimeFinder and Legato replace multiple tape-based backup systems with a standardized, managed backup-to-disk solution. The solution creates point-in-time copies of mission critical data and stores the copies on CLARiiON-based ATA drives, which enables simplified, rapid recovery. SRDF/A software performs asynchronous replication of data to an offsite data center.

Shattering business continuity myths

One of the common misconceptions about business continuity assumes the term is synonymous with disaster recovery. “Not so,” says Thomas of Austin Radiological Association. Disaster recovery relates to tools and technical glitches; business continuity plans must incorporate systems and procedures that enable an organization to recover from a disaster without long-term impact. For example, if a server in the hospital’s data center goes down, it isn’t necessary to implement the business continuity plan and turn on the offsite data center. Instead, the hospital replaces the server. But if a storm wipes out all of the data in the data center, the hospital with a robust business continuity plan can turn to its offsite data center, tap into redundant, replicated data and continue operations with minimal disruption.

“Staff will have preconceived notions about business continuity [and mission critical operations],” warns Wagner of El Camino Hospital. For example, at first glance payroll seems like it might rank fairly high on the system recovery priority list because the hospital wants to ensure staff are paid. This is a fallacy, says Wagner, because the department can work the previous run and reconcile after the system is up and running. Email, which seems like a low priority for system recovery, requires very stringent recovery time at El Camino Hospital. That’s because the email system is integrated with the VoIP message system, which may contain time-sensitive, patient care information. To further complicate the picture, each institution is unique; while there are commonalities among priorities, one hospital’s list of tier one applications may look very different from its neighbors.

The process of defining priorities and labeling applications as tier one, tier two or tier three can be cumbersome and politically charged. “Everyone’s application is mission critical to them, but it’s too expensive to put everything on a storage area network or replicate every single box. It’s important to work with upper management to determine what’s tier one, tier two, and tier three,” explains Laura Bagus, director of IT and infrastructure and telecommunications at Edward Hospital in Chicago. At the same time, the hospital does not need to define every single application at the outset of the plan. The hospital can use a single tier-one application like PACS as a foundation and build the business continuity plan that deploys multiple tiers from that starting point.

Another common misconception identifies business continuity as a project with an end date. Instead, it is an ongoing process that requires daily oversight as well as long-term planning and management, says Bagus.

Enterprise benefits

Butler Memorial Hospital in Butler, Pa., turned to EMC three years ago when it deployed PACS.  “We wanted to replicate imaging data in an automated fashion and ensure that images were easily and immediately accessible throughout the health system,” explains Cindy Esser, director of emerging technology.

The hospital’s PACS vendor recommended CLARiiON SAN as the cornerstone of the solution. The SAN holds one year of imaging data and facilitates instant access for radiologists and referring physicians. For the first year, a 600 DVD jukebox served as the backup. During the first year of PACS implementation, the hospital decided to evaluate its long-term storage and disaster recovery options.

After analyzing costs, the hospital concluded that continued reliance on the jukebox was more expensive than Centera content-addressed storage, says Tom Raraigh, radiology department director and PACS administrator. The drawbacks of the jukebox approach go beyond cost. The jukebox system requires more space—three jukeboxes—to hold the same amount of data as Centera, which uses a single standard rack. The jukebox requires manual interaction such as labeling, inserting and removing DVDs. Finally, it can take 30 to 60 seconds to restore an exam from the jukebox versus a few seconds with Centera. Given the advantages of content-addressed storage, the hospital opted to deploy the same technology for archived radiology images older than one year. 

“Centera is a truly hands-off operation,” states Raraigh. EMC Navisphere Manager centralizes management of CLARiiON arrays. Automated processes check storage integrity, and the system sends daily status emails to both Raraigh and EMC. The system’s self-healing architecture automatically phones the vendor in the event of a disk failure.

Another component of the Butler Hospital solution is a second offsite Centera system connected to the main data center via a fiber line. The replicated Centera environment protects patient data if a disaster occurs and enables HIPAA compliance. “We’re confident that we won’t lose data, which wasn’t the case with DVDs as they can degrade over time,” says Raraigh. The final element of Butler’s plan is EMC PowerPath software, which provides load balancing for data from mission-critical servers at all times.

Esser and Raraigh agree that the benefits of EMC solutions extend beyond tiered networked storage, encompassing rapid access and enabling HIPAA compliance. “PACS adoption would have been slower if we could not have provided immediate retrieval,” suggests Esser. Clinical care also is improved with rapid image retrieval enabled by Centera, says Esser. In other systems, it can take an hour or two to view older radiology studies, which can compromise clinical care.

Similarly, the transition to PACS at Edward Hospital and Health Services in Chicago provided the opportunity to implement a robust business continuity plan. Business continuity has become a business essential, says Bagus. “Being without data costs money, and it puts patients at risk,” continues Bagus.

Three years ago, when the hospital deployed PACS, it realized it required both short- and long-term storage with data replication. Direct attached storage no longer met the hospital’s needs; a SAN was necessary. So the hospital implemented a Symmetrix SAN environment in conjunction with its PACS as phase one of an ongoing business continuity plan. Another critical foundation of the plan was archiving. The hospital sends all PACS images to Centera for long-term retention.

While the Symmetrix and Centera storage platforms enable HIPAA retention requirements and have improved image access and retrieval to milliseconds, the hospital’s business continuity plan remains a work in progress. Under the ongoing plan framework, Edward Hospital tackles annual business continuity goals.

Two years ago, the hospital implemented the second phase of its plan and deployed its HIS ISB (Integrated Serverless Backup) on Symmetrix. The third phase of the plan is centered on a newly built outpatient facility that will incorporate a secondary data center for business continuity. In addition to redundant Symmetrix systems, the new site features AT&T Synchronous Optical Network (SONET) dedicated ring technology. The optical fiber network backbone ensures business continuity via a single fail-safe platform. EMC SRDF/S software provides real-time synchronous remote replication. 

The new center will provide a dramatic improvement over the hospital’s current disaster recovery plan. “It takes 10 to 12 hours to restore our HIS under the current program. With Symmetrix, we expect about 30 minutes of downtime in the event of a failure,” explains Bagus.

Conclusion

Healthcare’s reliance on IT solutions has increased dramatically in the last decade and delivered multiple benefits, including real-time, anytime-anywhere access to patient data, which in turn, accelerates care, trims costs and increases safety. Increased utilization of IT, however, brings new challenges. Healthcare IT environments encompass both mission and life-critical applications that require high availability and minimal recovery time in the event of system failure or outage. The enterprise cannot operate effectively, efficiently or safely if access to data is compromised during an outage. A number of proactive sites have opted to address the challenge by investing in and implementing a robust business continuity plan.

The essential ingredients of business continuity include storage, server, software and professional services. The plan may begin with the deployment of a mission or life-critical system such as PACS or CPOE, or it may develop from a strategic decision to implement business continuity. Regardless of the origin, an essential early task is selecting a primary partner that can deliver the storage, backup and recovery solutions as well as the expertise to implement the plan. Key internal players across the organization steer decision-making, develop performance level requirements and help segment applications into tier one, tier two and tier three categories to provide a framework for the various solutions that comprise business continuity.

Early adopters describe business continuity as an ongoing strategy and agree that a solid business continuity plan delivers multiple benefits. Implementing business continuity solutions provides the necessary insurance and peace of mind in the event of an outage or system failure and also meets HIPAA requirements. Although the insurance benefits are priceless, the advantages of business continuity transcend IT and regulatory realms. Many sites report that the decision to adopt business continuity makes more cost-effective and efficient use of the storage budget. For many sites, a SAN serves as the cornerstone of the business continuity plan. The SAN consolidates storage, which, in turn, enhances IT efficiency as staff are able to manage larger volumes of applications and data. At El Camino Hospital, for example, this approach could translate into $2 million in savings over 5 years.

A robust business continuity program serves as the backbone necessary to pioneer state-of-the-art imaging and IT solutions as well as keeping patients and patient data safe. The right plan can pay dividends in terms of time, money and resources—and get the business up-and-running quickly if tragedy strikes.