An IT Disaster Recovery Blueprint for Healthcare Organizations
“A Disaster? It Probably Won’t Happen to Me. Right?”
When you think of a disaster that can impact your healthcare organization, it’s easy to think of something rare and devastating on a mass scale. We think of disasters the media would cover—a massive hurricane, an F5 tornado, or California’s wildfires.
Thinking of a disaster as rare affects our ability to believe it won’t happen to us. Thus, it’s easy to not properly prepare for a devastating disaster, leaving you at risk for serious disruption.
For a moment, let’s think about a rare disaster. Why is it frightening? A few reasons:
- Potential injuries and loss of life.
- Serious property damage.
- Organizational disruption and possible operational shutdown.
- An inability to serve patients, risking their life and health.
- Destruction of IT systems and permanent data loss.
But what about the other reasons? Common disasters can easily cause organizational disruption, prevent you from caring for patients, and lead to permanent data loss.
It’s important to define a disaster by its impact rather than its nature. Before we comment on rarer disasters, let’s first look at disasters that could easily strike at any time.
What You’ll Learn in This Guide:
-
1
Common Disasters: Ransomware and Other Cyberattacks
-
2
Common Disasters: Flooding and Fire
-
3
Common Disasters: Power Outages
-
4
Seasonal and Geography-Specific Disasters
-
5
Disaster Best Practices
-
6
Monitor Your Backups
-
7
Endpoint Detection and Response (EDR)
-
8
Proactively Monitor and Manage IT infrastructure
-
9
Employee Training
-
10
Create a Disaster Recovery Plan
-
11
Disaster Recovery Checklist
Short on Time? Download the PDF! 👇
Common Disasters
Ransomware and other cyberattacks
When you think about a disaster, ransomware often doesn’t come to mind. It’s not caused by nature, it’s all electronic, and it seems like it’s just an IT problem if it happens. Yet, the impact of ransomware or another serious cyberattack can wreak havoc as much as or more than a natural disaster—seriously affecting your operations and finances.
Compare the effects of a ransomware attack with a natural disaster.
- Operational Disruption: Ransomware encrypts files and locks users out of their systems, effectively halting all operations until the ransom is paid or systems are restored. Prolonged downtime can occur, during which the healthcare organization cannot maintain access to critical patient information.
- Financial Losses: Some organizations may feel compelled to pay the ransom, which can be a substantial amount. (However, there's no guarantee that paying the ransom will result in the restoration of data.) Even if the ransom is not paid, the costs of restoring data from backups, cleaning systems, and rebuilding infrastructure can be significant. Failure to protect sensitive data can result in fines and penalties from regulatory bodies. Ongoing litigation will consume excessive time and resources.
- Permanent Data Loss: If backups are not recent or also compromised, some data may be permanently lost such as Electronic Health Records (EHR) and Personal Health Information (PHI). Even if data is recovered, it may be corrupted or incomplete, leading to further operational challenges. Hospitals and clinics may be unable to access patient records, affecting patient care and potentially endangering lives.
Municipalities are often targets for ransomware and other cyberattacks. Ransomware attacks get a lot of attention, but there are other cyberattacks that can also be devastating.
- Distributed Denial of Service (DDoS) Attacks: DDoS attacks overwhelm a municipality’s servers with traffic, causing service disruptions and potentially taking down websites and online services.
- Business Email Compromise (BEC): BEC attacks involve fraudsters impersonating executives or trusted partners to trick employees into transferring funds or sharing sensitive information.
- Malware: Malware includes various types of malicious software such as viruses, worms, and trojans designed to damage, disrupt, or gain unauthorized access to systems.
🔎 Related: Reality of Ransomware
Flooding and Fire
Flooding is the most common natural disaster that takes place in the United States, and major flooding events have increased during the last 10 years. In fact, 25% of flood insurance claims come from moderate- to low-risk areas—meaning a lack of past incidents are not predictors of future incidents.
Fires are also very common. Many originate through faulty wiring, overloaded circuits, defective equipment, poorly maintained HVAC systems, unattended cooking, cigarettes, improper storage of medical supplies, inadequate fireproofing of buildings, and other forms of negligence. Lightning strikes can also cause fires, particularly in areas with inadequate lightning protection systems.
Such common disasters can arrive out of nowhere, causing physical, operational, and financial devastation. From an IT perspective, consider the following repercussions:
- Infrastructure Damage: Floodwater or fire can cause structural damage to buildings, water pipes, electrical systems, machinery, and IT infrastructure such as servers, computers, and network equipment.
- Data Destruction: Physical damage to servers and storage devices can lead to the loss of critical data.
- System Downtime: Flooding or fire can cause extended downtime for IT systems, disrupting patient care and business operations.
- Recovery Costs: Data recovery efforts can be costly and may not always be successful, leading to permanent data loss.
Power Outages
Power outages are also a source of disaster, depending on the severity of the outage. From an IT respective, a major power outage can involve:
- Damaged hardware components from the sudden shutdown
- Unsaved changes to files and documents
- Data corruption caused by voltage spikes and irregularities
- Critical systems inaccessible due to a backup power system failure
Tornadoes
In the United States, tornado season lasts from approximately March through July, although a tornado can strike at any time. The South usually gets the most tornadoes between March and May, while the Northern Plains and Midwest are most at risk during June and July.
Remember, all it takes is an “average” tornado such as an F2 tornado, with winds upward of 150 miles per hour, to cause serious damage.
Hurricanes
Hurricane season lasts from approximately June through November, although a hurricane sometimes appears outside of that monthly range. States that most need to worry are the Gulf Coast states (Texas, Louisiana, Mississippi, Alabama) and East Coast states (especially Florida, Georgia, South Carolina, North Carolina, Virginia, and New York).
Even “mild” hurricanes can cause enormous flooding, power outages, and wind damage, from the coast to far inland.
Wildfires
Wildfire season lasts from approximately October through January, although wildfires can spread any time. While California gets most of the media attention for its massive wildfires, Texas actually experiences the most. The Southeast is also quite prone to wildfires.
There is often little warning before a wildfire tears through an area, giving people evacuation as an only option. The devastation to an organization can be apocalyptic.
Earthquakes
Thankfully, massive earthquakes are rare, but they can happen out of nowhere. Again, California is an obvious state at higher risk, but states such as Alaska and Hawaii also experience many earthquakes. Earthquakes can quickly destroy buildings—which contain your servers and IT equipment—and lead to permanent data loss and operational disruption along with loss of life.
Use onsite local data backups to lessen time to recovery for smaller incidents (such as a server failure).
An onsite backup solution can get you up and running again in minutes, which is crucial for minimizing downtime that can interfere with patient care. Local backups do not rely on an internet connection, ensuring data can be backed up and restored even during internet outages. IT staff can manage and maintain local backup systems directly, sometimes allowing for quicker troubleshooting and problem resolution.
For example, you might have an older server that dies after a severe power outage as a result of a bad thunderstorm. The onsite backup will take over and operate as that server until your healthcare organization can order and replace the hardware. Because there is a power outage that also prevents your connection to the internet, you are able to run the local backup once your generator starts working, allowing you immediate access to your data so that you can continue caring for patients.
Use offsite data backup to plan for worst-case scenarios. Offsite means storing your data backups far from your geographical location.
Offsite does not mean another floor on your building, or at another site within your metro area. It means geographically distant in case of disasters that take out all your local equipment. Flooding, hurricanes, and tornadoes can easily destroy backup servers onsite, leaving you with permanent data loss. And ransomware can infect all equipment on your network, including your local backups.
Storing backups offsite reduces the risk of losing all data due to a single catastrophic event affecting your primary location. By keeping backups in a different geographic location, you can mitigate risks associated with localized events, ensuring continuity of care. Plus, as long as you have an internet connection, you can remotely access your offsite data while restoration and repair takes place—allowing staff members to continue taking care of patients.
Offsite backups are also less susceptible to ransomware and other malware attacks that can encrypt or delete on-premises data. Data can be stored redundantly across multiple locations or cloud regions, providing additional layers of protection and availability. Anytime/anywhere access makes cloud backups an especially good choice for healthcare organizations, given the dire need to stay operational so that you do not risk lapses in patient care. (Remember, cloud applications still need to be backed up – many healthcare organizations assume anything in the cloud is automatically backed up, but that’s not the case.)
Monitor your data backups. It’s important to identify problems with your onsite and offsite backups before a disaster occurs.
Data backup monitoring ensures the integrity, reliability, and availability of your healthcare data when needed, confirming that backups are completed successfully without errors.
- Monitoring systems can send alerts in real-time if a backup fails or if there are issues. By detecting failures proactively, you can promptly troubleshoot and resolve problems before they lead to a significant disruption in patient care.
- Monitoring includes regularly checking the integrity of the backed-up data to ensure it is complete and uncorrupted. This guarantees that the data can be reliably restored. Don’t assume you’re able to resume normal healthcare operations after an incident, only to find out critical medical records or patient information is unavailable.
- Monitoring can help identify unusual activity that may reveal a ransomware attack or other security threats, allowing for immediate action.
Regularly test your data backups. If you don’t test your backups, you won’t know if you will be able to recover after a disaster.
It’s incredible how many healthcare organizations just assume their data backup solution is working. Then, they get a big shock when an actual disaster occurs and they cannot restore critical data. The result? Permanent data loss of important medical and patient information—when you thought you were doing the right thing.
Test. Test. Test.
We cannot say this enough. Don’t trust the backup dashboards or the reports your solution spits out. You won’t know if it actually works until you test it.
And don’t just check a few files and documents as a sample to reassure you. Everything critical must be restored and operational after a disaster—databases, healthcare applications, website, email, and documents. IT professionals can help you conduct a simulation that shows you how your data will look if restored after a disaster.
Ensuring that your data backup works is also a HIPAA requirement (164.308(a)(7)(ii)(A)): “Establish (and implement as needed) policies and procedures for responding to an emergency or other occurrence (for example, fire, vandalism, system failure, and natural disaster) that damages systems that contain electronic protected health information.” Failing to test means you are not ensuring that this requirement is followed.
Encrypt your backup data at rest and in transit—such as when you’re sending data backups to your data center or cloud provider. Make sure your decryption keys are stored both onsite and offsite.
Encryption ensures that only authorized users with the correct decryption key can access your data. This prevents unauthorized access to sensitive information such as medical records, patient data, and healthcare personnel information.
When transferring backup data over networks, encryption ensures that the data remains secure and protected from interception or eavesdropping by malicious actors. Similarly, encrypting your onsite backups prevents unauthorized access in case of theft or a malicious employee’s actions.
Encryption is also another important HIPAA requirement (164.312(e)(1)): “Implement technical security measures to guard against unauthorized access to electronic protected health information that is being transmitted over an electronic communications network.”
At first, this doesn’t appear to be a disaster recovery best practice. However, the inability for healthcare organizations to detect cyber threats has become a bigger issue over the last few years. Cyberattackers have become subtler and more sophisticated, often breaching and remaining undetected inside your systems for many, many months without your knowledge. Your response after detecting the cybercriminal can be the difference between proactively managing an inconvenience or reactively dealing with a significant cybersecurity incident that leads to permanent data loss and introducing serious risk to your patient care.
Antivirus is not enough anymore to protect yourself, and it’s become obsolete in lieu of EDR. This now-baseline tool uses machine learning (a form of AI) to detect anomalous behavior on your endpoints (servers, computers, etc.). The advantage of this tool is that the anomalous behavior doesn’t just have to be a virus or malware—EDR can detect healthcare applications used in strange ways, data exfiltration happening at odd hours, or ransomware in the act of deploying.
For example, if a threat is found on a computer, an EDR tool can cut that computer off from your healthcare organization’s network—preventing further spread of a dangerous virus that might shut down your operations. An EDR tool can be deployed easily, run in an automated fashion, and enhance the level of security protection for a healthcare organization at a low cost.
Like EDR, this is another best practice that doesn’t seem related to disaster recovery, but it lies at the root of many permanent data loss incidents at healthcare organizations.
Monitoring your hardware, software, and network equipment for issues lets you get ahead of problems before they become system failures and disruptions to patient care. For example, your IT team may see an alert indicating a high chance of server failure within the next six months. That allows you to take care of the problem today, avoiding an incident in the future where you might lose data. Continuous monitoring also helps detect vulnerabilities and potential security threats early, allowing for prompt mitigation.
Similarly, maintenance is important—especially in applying software patches and updates. Cyberattackers exploit security vulnerabilities all the time and they rely on many healthcare organizations not keeping up with software patching. Criminals don’t care if they take down a healthcare organization. Their impersonal scanning means that any organization is at risk—even if a cyberattack threatens people’s lives. Regularly applying security patches and updates helps not only protect systems from malware, viruses, and cyberattacks but also protects people’s lives and safety that can be put at jeopardy in the wake of a cyberattack.
Proper maintenance extends the life of IT assets, reducing the risk of hardware failure that leads to data loss and healthcare disruptions. Having up-to-date and well-maintained systems simplifies disaster recovery processes, minimizing data loss and recovery time.
Ransomware is bad enough—but you don’t want your healthcare staff to increase that risk. Yet, many employees are tricked by clever cybercriminals who use sophisticated social engineering tactics by means of emails that contain malicious attachments or links to malicious websites. Healthcare employees can be particularly susceptible because they are stressed, harried, and overwhelmed—increasing the risk of clicking on a bad link or attachment.
A cyberattacker’s strategy might involve posing as a trusted entity (such as a colleague, boss, or healthcare vendor) and convincing an employee to download a file or click a link. For example, an email might claim to be from IT support needing the employee to run a supposed update or security tool. Employees can also visit compromised or malicious websites that may trick them into providing their login credentials on fake login pages.
Cybercriminals often use these credentials to access networks and systems, where they deploy ransomware manually. They can also gain access to email accounts or other systems, spreading ransomware within the organization by sending infected emails from a trusted source.
Train employees to recognize phishing emails and understand the risks associated with opening attachments or clicking links from unknown sources.
🔎 Related: How Email Security Awareness Training Protects Against Phishing Scams
A comprehensive disaster recovery plan is essential for ensuring that your healthcare organization can quickly and effectively respond to and recover from unexpected disruptions. Your plan should identify potential threats (natural disasters, cyberattacks, hardware failures, human errors) and assess the potential impact of different disaster scenarios on critical healthcare functions and processes.
Objectives
Your disaster recovery plan should achieve the following objectives.
- Minimize downtime and the risk of delays when restoring impacted services.
- Protect and secure data, especially critical medical records and patient data.
- Ensure business continuity and continuity of care for patients.
- Guarantee the reliability of your data backup and standby systems.
- Minimize reactive decision-making during a disaster.
Components
Components of the disaster recovery plan include:
Assessing Critical Systems
Rank your applications and data by criticality.
- Recovery Time Objective: What is the maximum amount of time your healthcare organization can afford to have a system unavailable?
- Recovery Point Objective: How much data can you lose before it negatively affects patient care?
- Data Priority: Base your priority on the impact of not having a server or cloud solution functional.
- Low: Minor impact on staff and patients, and not critical to the daily functions of the healthcare organization.
- Medium: Some staff and patients affected, but not critical to the daily functions of the healthcare organization.
- High: All staff and patients affected, and critical to the daily functions of the healthcare organization.
Data Backup Policies and Procedures
Detail your data backup policies including:
- Frequency of backup: How often do you back up your data? Remember, patient records are updated frequently, necessitating continuous or very frequent backups to ensure data integrity and availability. At the same time, ensure that backup processes do not interfere with real-time access to patient data and critical systems.
- Type of backup: Decide what type of backup fits your disaster recovery objectives.
- Local backup: You have a server, device, or media that you use to back up data onsite.
- Remote Backup: Data is backed up to a remote server or data center.
- Cloud Backup: Data is backed up to a cloud-based storage service.
- Hybrid Backup: Combines local and cloud backup solutions.
- Storage Capacity: The volume of healthcare data can grow rapidly, necessitating scalable backup solutions. For example, medical imaging files (such as X-rays or MRIs) are large and require significant storage capacity.
- Retention period: How long are you legally required to retain your data? How long do you wish to retain your data? Most healthcare organizations must retain patient records for long periods (often decades) to comply with legal and regulatory requirements.
- Locations: How many locations need to be backed up? Many healthcare organizations have multiple sites, so this is an important component.
- Team: Define the disaster recovery team structure including team members, roles, and contact information. Clearly delineate the responsibilities of each team member and department involved in the disaster recovery process.
- Communications: Define how to communicate with employees, management, and the disaster recovery team during a disaster. Remember to also outline the communication plan for external stakeholders such as patients, partners, vendors, and regulatory bodies. Specify the communication tools and channels to be used such as email, phone trees, messaging apps, and emergency notification systems.
- Incident Response: Describe the steps to recover critical systems, applications, and infrastructure. When should the plan be implemented? Who can initiate the plan? What steps should be followed? What information needs to be logged? Important steps include:
- Acquiring replacement equipment.
- Restoring data integrity to the point of the disaster.
- Synchronizing backup data with any new data collected from the point of the disaster forward.
- Periodic Review: Review and update your disaster recovery plan periodically. Establish a regular schedule for testing the disaster recovery plan, such as quarterly or annually. Outline the process for reviewing and updating your plan based on test results, changes in your healthcare organization (such as opening new sites), or new threats.
- Documentation: Ensure that your disaster recovery plan is well-documented and easily accessible to authorized personnel. Include network diagrams, system configurations, and other technical documentation. Healthcare organizations must also be able to produce proof of compliance and data integrity for regulatory audits. As a result, you need comprehensive documentation of backup processes and data retention policies.
- Use onsite local data backups to lessen time to recovery for smaller incidents (such as a server failure).
- Use offsite data backup to plan for worst-case scenarios (such as a natural disaster or ransomware). Offsite means storing your data backups far from your geographical location.
- Monitor your data backups. It’s important to identify problems with your onsite and offsite backups before a disaster occurs.
- Regularly test your data backups. If you don’t test your backups, you won’t know if you will be able to recover after a disaster.
- Encrypt your backup data at rest and in transit—such as when you’re sending data backups to your data center or cloud provider. Make sure your decryption keys are stored both onsite and offsite.
- Use enterprise-grade antivirus and endpoint detection and response (EDR) to prevent and detect attacks.
- Proactively monitor and maintain your IT hardware, software, and network equipment. This includes software patching to eliminate cyber vulnerabilities.
- Periodically train employees about ways to spot phishing attacks and common cyberattacks. (95% of cyberattacks begin in an email.)
- Create a disaster recovery plan that clearly outlines how your municipality will recover your data and restore operations after a cyberattack or other disaster.
Is a disaster waiting to happen at your healthcare organization
Is a disaster waiting to happen at your healthcare organization? Implementing all these best practices may feel overwhelming.
VC3 helps prevent data loss by providing your healthcare organization onsite data backup for quick recovery after events like a server failure, offsite data backup for major incidents like a natural disaster or ransomware, real-time monitoring to quickly address data backup issues, and quarterly testing to verify your disaster recovery.
Have additional questions? Concerns about your current data backup and disaster recovery strategy? Contact us to talk with an IT specialist!