Learning from the SolarWinds Cybersecurity Attack to Build More Secure Systems

While we don’t yet know all the details, the recent SolarWinds Orion compromise brings to light a key tenant of software distribution and secure systems; namely that signing software release packages in a secure fashion is absolutely paramount. But it also helps us realize there is no magic bullet for security, and defense-in-depth must be realized at all levels of the software development lifecycle.

While the aftereffects of the SolarWinds compromise were devastating to many organizations, we can use what we learned to extract platform / vendor neutral concepts, and work through how they are used in all systems.

Also, while SolarWinds is a recent example of software signing gone awry, it’s by no means the only example. Numerous recent examples [1], [2], [3], [4] include various Linux/BSD environments in which similar techniques were used to GPG-sign packages and use mirrors to distribute the packages.

What do we know?

First, SolarWinds Orion management infrastructure was compromised. Second, digitally signed malware was introduced into the Orion update mechanisms. Third, the malware used a variety of techniques and methods on the targets to evade detection. Fourth, some amount of manual command & control was required. Finally, some easily guessable default credentials were used to access the SolarWinds platform.

We don’t explicitly know that the malware was signed with the SolarWinds keys and certificates. However, let’s walk through that scenario as a way to discuss the need to sign and update applications in a secure fashion.

What do we mean when we say a library or application was digitally signed?

Digitally signed implies that some form of a cryptographic hash was generated for a component and that this hash was cryptographically signed (likely using a public/private key pair). At runtime, the signature is verified against a database to ensure the software or library originated from a trusted source. Once the signature has been verified, the cryptographic hash of the software or component is regenerated and compared to the signed hash to ensure it wasn’t changed or modified.

Code signing is generally handled with the use of public/private key pairs and uses algorithms such as RSA or ECDSA. The private key is held in confidence by the signer (i.e. the software developer) and the public key is distributed with the operating system or installation software. Public keys themselves can also be signed (therefore providing a web of trust) with various mechanisms for checking a certificate’s validity, mostly to ensure it has not been revoked by the developer or the OS vendor. As alluded to above, GPG is a specific protocol or instantiation of public/private keys and includes various bits to revoke keys, build a web of trust, etc.

Getting malware signed

It’s no surprise that, with a switch to digitally signed software components, we’re seeing an increase in (properly) signed malware [5]. Malware may be signed with everything from stolen certificates to forged hashes to using a vendor’s continuous-integration or release systems to perform the signing of modified binaries. There are, of course, variations in all of these as well, such as finding a way to inject modifications into a release pipeline such that they become inherently trusted or signed, compromising a certificate signing authority, or acquiring a certificate for defunct vendors and software.

With SolarWinds Orion utilizing easily guessable credentials, we can make some educated guesses about the path used to digitally sign the malware distributed through the platform.

1. Gaining direct access to the continuous integration/release platform.

2. Using the credentials to gain access to the release system and using that system (and any stored credentials, tokens or trust mechanisms) to move laterally within the SolarWinds network to gain access to the release and/or staging systems.

3. Injecting or modifying artifacts into the staging or build/release pipelines.

4. Hash collision.

Working backwards, a hash collision is likely not the culprit here as for the most part, hash algorithms have been updated to be more collision resistant. That’s not to say that it wasn’t possible if an older hash algorithm (such as MD5 or SHA1) was being used. But even if that were the case, requires a bit of guesswork (if not pure luck) to find a hash that matches an existing application or library.

We can effectively consider the other approaches to be nearly identical as they really only differ in the logistics.

Again, we don’t actually know that SolarWinds certificates were used to sign the malware. However, it seems it was properly signed, meaning an approved authority signed it (or the same techniques highlighted above were used to gain access to another parties’ signing keys). The resulting signed binaries were then distributed using the SolarWinds infrastructure. Either way, it gives us a lot to consider in terms of Secure Software Development AND Delivery.

There is, of course, more that could be explored here (i.e., lateral movement, injecting artifacts or modifications into a staging or release area, etc.), but these seem like good candidates for a future series of articles.

Wait, so how do we sign release packages securely?

Breathe deep and brace yourself because, the short answer is, “it depends”.

There is no single magic solution for secure release management that works for all developers, all environments, and all use cases. What is evident however is that you have to choose a process that works for your environment and enables you to make the appropriate tradeoffs.

One possible approach to securely signing software uses an offline approach combined with some form of hardware security module (HSM). Within this model, the software is developed and transferred to a staging area. In the staging area, the software is signed using a non-extractable key (presumably by a trusted individual within your organization). The software is then transferred back to the release / deployment environment.

This process could be made more secure by requiring the key stored within the HSM to use some kind of multi-factor authentication. This would then prevent someone who had access to the HSM from being able to sign their own packages. It doesn’t, however, directly prevent someone from modifying or injecting build artifacts into the development environment before they are transferred to the staging environment. In order to address that threat, we then have to look at our entire DevSecOps pipeline, which could be covered in its own series of articles…

In short, securely signing release packages (and verifying the contents of those packages before signing) is no small feat but is a major, though often overlooked, part of software development and software release.

References

1. https://duo.com/decipher/flaw-in-apt-utility-allows-malicious-package-installation

2. https://www2.cs.arizona.edu/stork/packagemanagersecurity/attacks-on-package-managers.html

3. https://www.researchgate.net/publication/221609207_A_look_in_the_mirror_Attacks_on_package_managers

4. https://www.zdnet.com/article/microsoft-spots-malicious-npm-package-stealing-data-from-unix-systems/

5. https://blog.trendmicro.com/fake-antivirus-solutions-increasingly-stolen-code-signing-certificates/

---