How Sky Lakes Medical Center Overcame a Ransomware Attack

The Oregon hospital's director of information systems talks about lessons learned from a 2020 attack by Russian hackers that forced executives to shut all systems down.

Ransomware attacks are a serious threat to healthcare organizations, and nearly every hospital has a strategy in place to deal with one if it happens. But those who've experienced such an attack say even the best-laid plans go awry, and the best strategy is to expect the unexpected.

"We had downtime processes that worked very well for the first 24-48 hours," says John Gaede, director of information systems at Oregon's Sky Lakes Medical Center. "And then they all broke down. We had to invent a lot of what we did in the moment."

Located in Klamath Falls, near the California border, Sky Lakes Medical Center nestles alongside Klamath Lake, surrounded by forests in a high desert region 60 miles south of the Cascades. The area is a popular recreation destination, yet in October 2020 that calm was shattered by a group based halfway across the world in Russia.

According to the FBI and Health and Human Services Department, the hospital was one of a dozen attacked at the same time by Ryuk ransomware threat actors, a group known for being able to change methods on the fly to evade detection. The attack lasted roughly three weeks, though the recovery process took a lot longer and prompted Gaede and his colleagues to re-examine their protocols.

"We've gone to every single department to document what was done before we forget this," he says. "We haven't even finished that playbook yet, but we've learned some valuable lessons."

John Gaede, director of information systems, Sky Lakes Medical Center. Photo courtesy Sky Lakes Medical Center.

Among them, Gaede says: Make sure your partnerships with tech vendors are strong and lean on them for help. And be prepared to be surprised and versatile enough to react to those unexpected effects.

"It's people, processes and technology," he says.

The attack occurred just after noon on October 26, 2020, when an employee clicked on an e-mail with a link that supposedly discussed a bonus (not an unusual or unexpected e-mail, Gaede says). The file was downloaded from Google Drive, and at the same time the employee's computer blinked, and the employee rebooted the computer. Nothing seemed out of the ordinary, so the employee didn't alert anyone.

By 11 that evening, Gaede says, the first encryption efforts were conducted on Windows-based systems, and soon after all systems were slowing down. At 3:30 a.m., he got the phone call from IT alerting him of the ransomware attack.

Suddenly it was all hands on deck. Gaede says they turned to their Vocera communications platform to restore functionality, but everything there was encrypted within minutes.

"We then realized we had to shut everything down," he says.

Now, "shutting down" is a terrifying thought. The 176-bed hospital had to go offline immediately, taking down more than 2,500 PCs and 600 servers and halting some clinical care services, alongside all of the connected care aspects of a hospital serving roughly 120,000 people in a relatively remote area. Even maintenance and environmental services were affected.

This is where the best-laid plans often break down. Hospitals can train their personnel on what to do in the event of a ransomware attack, going through a number of different scenarios and if-this-then-that situations, but eventually the ramifications prove too complex. It's one thing to map out all the results of putting the EHR platform into a downtime mode, and quite another to understand how that affects business operations such as supply chain, pharmacy, and revenue cycle.

For instance, just as the snow was beginning to come down in Oregon, Gaede suddenly found out they didn't have heated sidewalks.

"At this point, every system is not happening," he recalls. "We didn't know what had been compromised. We realized this wasn't going to be easy, and it wasn't going to be like we had planned."

Gaede says hospital executives huddled quickly that morning and then went into action. Asante, the health system located some 70 miles distant in Medford, Oregon, was alerted. The hospital's insurance carrier was contacted, as well as Cisco, which sent in its Cisco Talos. In short order, both Cisco Talos and Kivu Consulting were helping with recovery efforts.

"We had to completely rebuild our network," Gaede says. "That's a laborious process. We had to build backups and test them first to make sure they were clean, then run the [main systems] through tests to validate that they can work. We didn't want to start something up, have it [integrate] with another system and have everything fall apart."

Ironically, doctors and nurses who had spent the last 18 years getting used to the EHR platform now had to go back to the old way of doing things. Bar-code scanning was out, as was typing data into the system, or even jumping online to track down information or do research. Copper line fax machines were back in vogue—though one literally blew up from overuse.

Everything was now written on paper or spreadsheets, and data and messages were conveyed from one department to another by runners, in what came to be called the 'sneakernet.' The hospital ran out of prescription pads, and local retail outlets were swamped with requests for paper.

"Few of them had been trained on how to do medicine on paper," Gaede says. "We went to Walmart [and] Staples and bought all the paper they had."

The hospital also had to shut down its PACS system, and for several days couldn't provide any imaging. Gaede says they contacted Sectra/Electromek, which came in and built a whole new system on the AWS cloud, so that images could be read on an iPad by the following Friday (the RIS was enabled by Saturday). 3M M*Modal then came in to integrate their services for reporting.

On November 9, radiation oncology was back up, and on November 23, the hospital restored its Epic EHR platform, alongside a fully operational PACS and radiology system. While things certainly weren't "normal" at that time, Gaede says everyone in the hospital was able to take a deep breath and relax a bit.

"It was like a brand-new go live," he says.

He says some clinical care was affected, though it's hard to translate that into clinical outcomes. Some services were delayed, some patients were inconvenienced, and Sky Lakes executives, clinicians and staff were all put through the ringer.

The hospital's revenues and cash flow were also affected, forcing executives to dip into cash on hand to make sure everyone got their paychecks. Gaede says the incident will affect their business plan for about three years, with more money invested in security and data protection.

"We were just trying to take care of our community, and we had no notion that state actors were taking aim at us. The potential impact to patient care and patient harm was absolutely real," he says.

But the hospital's disaster plan held up, for the most part. Legacy backups from Cohesity worked as they were supposed to, staff knew what they had to do or they knew how to react when they didn't know what to do. Requests for help were answered, and no ransom was paid.

"We made it a point right from the start that we would be transparent about what happened," Gaede adds. "It was very important to us. And I can look back now and feel we did the right things. Hopefully this will help others in the future."