In An Instant: Disaster Recovery & Business Continuity

As digital medical images continue to gain favor, healthcare organizations face new challenges that didn't exist when images were solely analog. And due to increasing pressure from government and industry regulators and the shock of a attacks, blackouts and fires, two of these issues - disaster recovery and business continuity - have risen to the top of the list of healthcare IT managers' concerns.


DISASTER RECOVERY OR BUSINESS CONTINUITY: WHAT'S THE DIFFERENCE?

Disaster recovery (DR) and business continuity (BC) are often treated interchangeably, but in fact, the two terms have distinct meanings. Disaster recovery refers to an organization's ability to maintain the integrity of its technology infrastructure, applications and data in the event of a disaster. Disaster recovery involves data and system redundancy and their accessibility following hardware or software failure or destruction. In medical imaging, this includes issues such as accessibility of patient images; availability of a picture archiving and communications system (PACS), radiology information system (RIS), or image acquisition equipment; and reliability of the radiology or hospital network.

Disaster recovery is usually viewed as a single component of business continuity, which is the ability to withstand business interruptions that might occur as the result of a disaster or some other abnormal incident. Besides disaster recovery, business continuity also includes maintaining everyday functions that contribute to smooth business operations, such as workflow, scheduling, staffing, and communications. Business continuity concerns in medical imaging include availability of IT staff, maintenance of communication networks such as email and phones, and provision of consistent workflow.


DISASTER PLANNING: BE PREPARED

An organization's best ally in disaster planning is a robust disaster recovery/business continuity plan. And courtesy of a regulatory environment that demands their preparation and implementation, such a plan is no longer just an option for healthcare organizations.

As you probably know, the Joint Commission on Accountability in Healthcare Organizations (JCAHO) requires U.S. healthcare organizations to develop business continuity plans that include provisions for patient data security and loss protection; analysis of potential disasters; and establishment of emergency communications plans.

In addition, the Health Insurance Portability and Accountability Act (HIPAA) security rule, which will become effective in April 2005, mandates data backup and disaster recovery plans for all healthcare organizations.

"Because of HIPAA regulations, hospitals are very interested in [business continuity]," says Steve Higgins, director of business continuity for storage company EMC Corp. "The big challenge is [having] a comprehensive business continuity plan that addresses how the different technologies are connected and ensures highly backed up and available data."


BUSINESS CONTINUITY PLAN PRIMER

There are several steps in creating a viable business continuity plan.

Organize the planning team. A healthcare facility's executive management should endorse the project and be heavily involved in the planning team. Depending on the size of the organization, members could include the CIO, radiology administrator, radiology department head, IT project manager, PACS administrator, and director of radiology informatics. An experienced project manager should be chosen as team leader.

The project manager should make sure that team members understand HIPAA and JCAHO security rules. The team should create goals, objectives, and project milestones and be held accountable for achieving these goals.

Establish the most likely disaster scenarios facing your institution. Understanding the types of potential disasters that could be faced by an organization can help IT managers develop and test scenarios for business continuity and disaster recovery plans in their imaging departments.

At the Cleveland Clinic, business continuity scenarios were ranked in the order in which they were most likely to happen. "We dealt with the more common failures and then worked down the line to the less common failures," says David Piraino, M.D., section head of computers in radiology at the Cleveland Clinic.

According to Piraino, the Cleveland Clinic's business continuity team determined that the most common failure in imaging IT is workstation loss. "We try to deploy workstations so that there's more than one workstation in any environment unless there's no choice," he adds. "That way if there's a failure on a single device, we can still function because there's another device as a fallback."

The University of California at San Francisco (UCSF) chose to evaluate disaster scenarios based on the expected amount of downtime, according to Katherine Andriole, Ph.D., associate adjunct radiology professor and PACS clinical coordinator at UCSF's Radiological Informatics Laboratory. "For example, a power outage from a rolling blackout is treated as a short-term crisis," she explains. "We have scenarios for disasters that would have a longer term effect, such as earthquakes, but - knock on wood - we haven't had to use them yet."

Determine the impact of each type of disaster on your business. Often seen as the most difficult part of disaster planning, business impact analysis helps the planning team understand the types of losses an organization might face in the event of a catastrophe. For example, a facility may experience a loss of data and systems; the loss of access to data and systems; or the loss of staff. The response to each type of loss will be different.

In addition, determine your organization's mission-critical functions, the maximum amount of time they can be non-functional, and the additional number of staff that would be needed to compensate. Consider also the issue of equipment replacement. How much would it cost and how long would it take to replace damaged equipment?

Another factor your planning team will need to consider is the location of the malfunction. A recovery effort in the radiology department is different from one in the emergency room or an outpatient clinic. And an interruption in web-based access presents its own set of unique problems.

Develop the plan. Based on the knowledge gained by assessing business impact and understanding disaster scenarios, the planning committee should put together a plan that provides specific responses for each type of disaster. The team should address failure of mission-critical processes; loss of systems, assets and employees; and interim workflow scenarios. As with most technology decisions, the specifics of each plan will vary depending on the needs and budget of the healthcare facility.

Some examples of items that might be addressed in a business continuity plan:

• Non-emergency patients. Some hospitals may choose to cancel all non-emergency outpatient appointments during an influx of disaster victims.
   
• Backup power. An institution may decide to keep its imaging equipment connected to backup power, but not its air conditioning.
   
• Emergency staffing. During a network outage, all IT staff may be required to report to the hospital.

In addition to interim business continuity measures, the final plan should also include strategies for disaster prevention as well as long-term recovery.

Test and maintain the plan. Periodic testing and revision of the plan is required under HIPAA's security rule. Testing allows the planning committee to observe and correct any gaps in the plan, and provides staff with disaster training. The plan should be reviewed and updated periodically to account for organizational changes.


TECHNOLOGY SOLUTIONS FOR SEAMLESS DISASTER RECOVERY

Planning for recovery of imaging data and continuity of a healthcare facility's imaging business is a critical part of its broader disaster recovery/business continuity efforts, says Jocelyn Young, program manager for U.S. vertical industry research at market intelligence firm IDC. "Imaging data are the same as any other patient data, except the size and volume of the files is so much larger," she adds. "On the surface, it's all patient information but there are some underlying technology concerns when it comes to backing up large volumes and large files."

According to Young, the primary issues are determining and maintaining the appropriate storage capacity and managing the data cost-effectively. In addition, hospitals must decide when to move their imaging data from short-term to long-term archives - a decision that depends on both its budget and the volume of data the facility generates.

In the not-so-distant past, disaster recovery meant using tape backup to archive large volumes of data. While still an option, tape backup isn't automated, doesn't provide continuous data access, and doesn't protect systems and applications. Fortunately, newer network-based solutions such as storage area networks (SANs) and network attached storage (NAS) provide automated, instantly accessible backup of mission-critical applications and short-term and long-term archives.

Depending on the size and output of the facility, short-term archives can include between two weeks and two months of patient data, while a long-term archive might hold records and images more than a month old. Images can be automatically sent to either or both archives.

Many facilities implement plans for redundant hardware systems to ensure network continuity in the event of a disaster at the site. After all, data and applications are useless if there's nothing to run them on. Redundant networks can be outsourced via an application service provider (ASP), or they can be housed at a second site such as a sister hospital or an outpatient clinic. Some equipment vendors provide options for quick shipment of replacement equipment, while others offer insurance that guarantees the equipment will be replaced in a certain amount of time in the event of disaster. And for organizations with bigger budgets, vendors will reserve fully- or partially-redundant offsite space that can be used in case of a crisis.

How a facility approaches equipment redundancy will depend on its budget, volume of data, and the severity of the disaster scenarios in its business continuity plan.

According to Doug Chandler, program director for storage and data management services at IDC, healthcare IT is still developing the best strategies for managing disaster recovery and business continuity. "Some organizations may look at this and decide there's no possible way of managing this internally, and [they] outsource to a third party," he says. "Others may think there's too much sensitivity in terms of patient information and they can't possibly outsource. It's a build vs. buy issue."

The good news is that storage capacity is becoming less expensive, says Chandler. "On the flip side, it isn't getting cheaper to manage tons and tons of data yet," he adds.

Many organizations find that the most convenient time to upgrade their data archiving solution is when they purchase or upgrade their PACS. A PACS vendor may have their own DR/BC solutions or they may partner with top storage vendors to offer solutions. "Storage needs to be a clear part of PACS. The leading [PACS vendors] are aware that storage has to be a baked-in solution," says Jon Mello, EMC's director for global healthcare solutions. "You can't have one without the other."

The bottom line? If a PACS vendor isn't prepared to discuss options for DR/BC, show them the door!
 

Case Study: Brigham & Women's Hospital

With 700+ beds, Boston-based Brigham & Women's Hospital (BWH) has a complex image archive that includes data from two partners - the 150-bed Faulkner Hospital in nearby Jamaica Plain, and the Dana-Farber Cancer Institute, a specialty clinic. BWH easily produces two terabytes of imaging data per month, according to Ramin Khorasani, M.D., medical director of information management at BWH's department of radiology. "No other organization has this much of a data-generation headache!" Khorasani says.

BWH's disaster recovery and business continuity plans required an enterprise-wide archiving system that would operate in a multi-site, multi-vendor environment that includes ten imaging departments, approximately 150 image acquisition devices, all imaging modalities, and an integrated EKG database. For short-term access, BWH uses EMC's CLARiiON, which archives between 10 months and one year of data. Data that are older than that are in the process of being migrated from tape to EMC's Centera, which is a solution for fixed data - data that no longer need to be modified.

BWH's objective in the event of a disaster is to have near-instant access to all of the data, not just the data archived on the short-term solution. "All of our data, going back to our September 1998 PACS installation, is going to be online," says Khorasani. "Our goal is to have rapid access to all of it."

Because imaging data is archived automatically at both BWH and the Faulkner Hospital, the organization is assured of redundant data, applications and systems. As of this writing, the system is untested - it's been deployed for only about six months. "We haven't had a crisis yet, and I don't want to!" says Khorasani.