Pacific Northwest National Laboratory CIO Jerry Johnson takes you inside the cyber attack that he faced down–and shares his security lessons learned.

By John Foley InformationWeek

When Pacific Northwest National Laboratory detected a cyber attack–actually two of them–against its tech infrastructure in July, the lab acted quickly to root out the exploits and secure its network. PNNL then did something few other cyber attack victims have been willing to do. It decided to talk openly about what happened.

The lab’s CIO, Jerry Johnson, last week provided a detailed accounting of the cyber attacks. Speaking at the IW500 Conference in Dana Point, Calif., Johnson described how intruders took advantage of a vulnerability in one of the lab’s public-facing web servers to plant a “drive-by” exploit on the PCs of site visitors, lab employees among them. For weeks, the hackers then surreptitiously scouted PNNL’s network from the compromised workstations.

Simultaneously, a spear-phishing attack hit one of the lab’s major business partners, with which it shared network resources. This second group of hackers was able to obtain a privileged account and compromise a root domain controller that was shared by the lab and its partner. When the intruders tried to recreate and elevate account privileges, this action triggered an alarm, alerting the lab’s cybersecurity team.

Within hours, the lab made the decision to disconnect its network in order to sever the hackers’ communications paths and contain any further damage. Over the July 4 weekend, while the rest of us were grilling burgers, PNNL’s security team conducted cyber forensics, reconstructed the domain controller, re-imaged systems, and restored network services that had been taken off line.

Who was behind the attacks? That’s one question CIO Johnson won’t discuss. But it’s worth noting that Dept. of Energy facilities were reportedly targets in the series of cyber attacks known as Operation Shady RAT that were carried out against more than 70 companies, defense contractors, and government agencies over the past few years. Based on the available evidence, some experts have speculated that those attacks originated in China.

At the IW500 conference, in a session titled “Anatomy of a Zero-Day Attack,” Johnson was candid about how the lab responded to the intrusions. He also shared the following list of lessons learned from the experience:

1. There’s danger in multi-level security environments. The lab had a well-protected IT security perimeter, but the attacks made it through anyway. An advocate of “defense in depth,” Johnson is putting increased emphasis on protecting the data itself.

2. Purge legacy, minority technologies. The Web server in the first attack was based on a little-used technology at the lab, Adobe ColdFusion. Such out-of-sight, out-of-mind technologies are inherently vulnerable because they don’t get the same degree of attention as an organization’s primary platforms.

3. Monitor cybersecurity events 24 x 7. Advanced persistent threats like those that hit PNNL are just that–persistent–and require constant vigilance. Across federal government, agencies are investing in “continuous monitoring,” with a goal of obtaining a near real-time view into the status of computer system security.

4. Maintain a core forensics capability. If your network does get hacked, security teams must be able to reconstruct events and assess the damages. What you learn can help prevent a relapse.

5. Include a senior project manager on your response team. Responding to a breach requires not only attention to detail and carefully coordination, but an ability to engage top management at a moment’s notice and, if necessary, escalate decision making.

6. Be prepared to call for help, and don’t wait. You may need to bring in security experts, business partners, law enforcement, or other outsiders. At PNNL, Johnson alerted the public affairs office, in order to prepare for the inevitable media inquiries.

7. Have an emergency communications continuity plan. When PNNL pulled the plug on its network, the hackers lost their ability to inflict further damage. Unfortunately, the decision also meant that lab employees lost network services, including email and voice mail. Be prepared for that eventuality by sharing cell phone numbers and alternative email address in advance.

As Operation Shady RAT and a similar cyber attack on Google and other companies demonstrate, the risks are complex and growing. Johnson agreed to talk about it as a way of helping other organizations bolster their defenses. For that, he deserves a tremendous amount of credit. Secrecy is the norm in the wake of a cyber attack, but openness will lead to better preparedness.

John Foley is editor of InformationWeek Government.