On Sunday, we learned that federal agencies and other organizations had been penetrated by nation-state attackers, identified as Russian by multiple sources.
Though the definitive attribution for the attacks won't come for a while, we do know how the attack occurred.
SolarWinds Orion, a widely used network monitoring tool, had been compromised. In what's known as a "supply chain attack," the attackers injected malware into the update code last March, and customers had been installing a Trojan each time they ran an update.
"Supply chain attacks are low-cost, high-impact threats," said Kelvin Coleman, executive director at the National Cyber Security Alliance.
That obviously makes them particularly attractive for bad actors, he told DCK, since they can reach many targets simultaneously. "The malware-laced update introduced through SolarWinds’ Orion products became a single disruption point for companies that thought they were simply installing or updating verified software."
As many as 18,000 organizations were affected, SolarWinds said in an SEC filing on Sunday. The company claims to have more than 300,000 customers total, including all ten of the top ten US telecommunications companies, all five branches of the US military, all five of the top-five US accounting firms, the Pentagon, the State Department, the National Security Agency, the Department of Justice, and the White House.
The company also said that more than 425 of the US Fortune 500 were customers, including Ford, Kodak, Cisco, Mastercard, Staples, and Microsoft.
After the news broke, SolarWinds took down the page with its customer list.
The SolarWinds Orion collection of IT monitoring services is widely used in data centers, said Mike Lloyd, CTO at RedSeal, a cybersecurity company. "Essentially, anything you run in a cloud or physical data center needs to be monitored, to track uptime, performance, and general service availability," he said.
These monitoring tools tend to be set up to pay the most attention to the most critical assets, he added. "This is why SolarWinds Orion is such an important target for espionage. If you can see what it can see, you can effectively see everything important."
After installing the Trojan, the attackers used it to penetrate some of their victims' networks and exfiltrate sensitive data and emails.
Confirmed cases include the Commerce Department, the Treasury, and the well-known cybersecurity firm FireEye.
It was the FireEye attack that was the hackers' undoing. FireEye spotted the intrusion and was able to stop it before the attackers got anything more sensitive than a set of open-source penetration tools. FireEye then went on the offensive to determine how the attack happened and who else might have been affected.
FireEye said it detected intrusions at multiple entities worldwide, including government, consulting, technology, telecom, and mining organizations in North America, Europe, Asia, and the Middle East.
"We anticipate there are additional victims in other countries and verticals," the company said. "FireEye has notified all entities we are aware of being affected."
This isn't be the first time Russia used a supply chain attack with devastating results. In 2017, Russian military intelligence hacked the Ukrainian accounting software MeDoc to launch the NotPetya malware around the world – resulting in an estimated $10 billion in total damages.
How the Attack Worked
The SolarWinds attack – dubbed Solorigate by Microsoft and Sunburst by FireEye – starts out when an enterprise downloads an update.
As it turns out, SolarWinds recommended that enterprises exclude this update process from anti-malware checks.
The SolarWinds Orion tool helps enterprises monitor their networks. A compromise of the system gives attackers access to the entire network infrastructure.
The malware communicates with its makers via communication servers set up using legitimate domains the attackers purchased, which have existed for a while. That allowed it to evade defenses looking for suspicious traffic to known malicious sites or brand-new domains.
"After an initial dormant period of up to two weeks, it retrieves and executes commands... that include the ability to transfer files, execute files, profile the system, reboot the machine, and disable system services," FireEye said in its report.
The malware disguised its activity as legitimate Orion Improvement Program traffic and stored its reconnaissance results in legitimate plugin configuration files. It also used obfuscated blocklists to identify anti-virus and other security tools.
The attackers then forged identification security tokens that allowed them to impersonate any user or account, including privileged accounts, which allowed them to bypass multi-factor authentication for services like Office 365 and get into email accounts, both on premises and in the cloud, from any vendor.
"They are highly trained in operational security and executed with discipline and focus," said FireEye CEO Kevin Mandia in a statement. "They operated clandestinely, using methods that counter security tools and forensic examination. They used a novel combination of techniques not witnessed by us or our partners in the past."
The attackers then used their access to infiltrate existing user accounts or create new ones in order to access more systems.
Potential Impact
As of publication date, the list of known victims includes FireEye, an unnamed US think tank, the US Treasury Department, the US Department of Commerce, the National Institutes of Health, the Cybersecurity and Infrastructure Agency, the Department of Homeland Security, and the US Department of State.
The full list of victims could be much longer.
"It’s a humbling but important realization that you may never be able to tell whether you were actually compromised, only whether or not it was possible," said RedSeal's Lloyd. "If it was possible, it’s best to assume it did happen."
So far, the goal seems to have been to exfiltrate information without being destructive, and none of the victims have reported damage to their systems.
For government agencies, contractors, and other organizations with sensitive information, however, the loss of sensitive information could be devastating.
As a result, DHS's Cybersecurity and Infrastructure Security Agency issued a rare emergency directive on Sunday ordering federal agencies to shut down all their SolarWinds Orion tools immediately and review networks for indications of compromise.
“The compromise of SolarWinds’ Orion Network Management Products poses unacceptable risks to the security of federal networks,” CISA Acting Director Brandon Wales said in a statement.
But that doesn't mean that other types of organizations should feel safe, said Scott Crawford, analyst at 451 Research.
"It's easy for any organization to underestimate the extent” of interdependencies between different vendors serving the enterprise IT market, he told DCK. "You might think you're a niche supplier serving a small segment of the market, but the people who use what you provide can have their own dependencies that are substantial."
For example, the NotPetya attack cost shipping giant Maersk an estimated $300 million. "The dependency there was on one provider of one set of business software that operated in one geographic locale: Ukraine," said Crawford. Maersk was collateral damage in a cyberwar between Russia and Ukraine.
The US is involved in a similar war, a cold cyberwar with a constant undercurrent of below-the-surface attacks – one in which any company could become collateral damage, he said.
The fact that a compromise was developed by nation-state actors doesn't mean that non-state cyber criminals can't take advantage of it.
"The boundaries between threat actors can be indistinct sometimes," said Crawford. "If they share common objectives, there may be good reason for them to cooperate."
Recommended Responses
The first step any data center security manager should take is find out to what extent their infrastructure is affected.
SolarWinds posted instructions for determining which version of the Orion product a data center might have running.
Then, CISA recommends saving an image of operating systems and system memory for each case where a vulnerable SolarWinds product was running, analyze for new user or service accounts, and inspect stored network traffic for indicators of compromise.
FireEye has also posted lists of indicators of compromise and shared a GitHub repository of free countermeasures.
The next step is to remove, upgrade, or isolate the affected SolarWinds Orion systems. CISA recommends complete removal, while SolarWinds recommends upgrading and patching.
In those instances where an upgrade or patch is not immediately feasible, SolarWinds recommends isolating the system as much as possible.
Recommendations include running Orion behind a firewall, disabling its internet access, and limiting ports and connections to only those that are absolutely necessary.
Next, enterprises need to remove all compromised user accounts and communication channels, including shutting down all access to known compromised domains and IP addresses.
All credentials that the SolarWinds systems had access to should be reset, as well as all privileged accounts and authentication keys and tokens.
Sophos put out an extremely detailed incident response playbook for the SolarWinds compromise and has been continuously updating it as new information becomes available.
After the cleanup is completed, organizations can restore systems to a last known good state or reinstall from trusted sources.
Microsoft has also released a detailed SolarWinds response advisory.
Prepare for the Future
The SolarWinds hack was a "watershed event," said Luke Tenery, partner at StoneTurn, a global advisory firm specializing in regulatory, risk, and compliance issues.
"It showcases how even the most advanced organizations, even when likely taking some level of precaution, cannot detect highly advanced attacks when trusted third parties have been compromised," he told DCK.
In this case, a leading security company fell victim, as well as some of the most security-conscious parts of the US government.
That's the big danger of supply chain attacks, he said. Organizations put trust in their IT vendors.
The solution is to enforce vendor risk management, he said. "But to also monitor your systems for integrity and expected behavior – trust but verify – even for the most prominent technology providers in the US.
