The Most Secure Linux for Embedded Systems – Hint, It’s Not Enterprise Linux
Don’t get upset.
We fully acknowledge that the hard-working people who make enterprise type Linux distributions want to help their customers achieve secure systems and they strive to improve the security posture of their distributions all the time.
However, there are two major areas where these developers run up against a nearly insurmountable wall.
First, they use a different threat model. An enterprise distribution of Linux is designed with a threat model suited to datacenter/cloud deployments, but embedded and cyber-physical systems face a wider array of threats including situations where the attacker may have physical access to the system.
Second, enterprise systems do not follow a secure by design philosophy. If you don’t believe me, consider this 120-page document on how to secure your distro. A Linux distro that was secure by design wouldn’t need nearly this level of explanation as most of the security would be built-in and out of the customers’ hands. Helping customers through this process and adding our products to address remaining vulnerabilities is a significant source of work for us.
As such, let's look quickly at the specific ways these "philosophical" differences in security mindset result in different security outcomes. In particular, let’s examine how enterprise Linux distributions fare when measured against the 10 Principles of Secure Embedded Systems.
The Good (Enterprise Linux does get a few things right…)
Mandatory Access Control: Modern Enterprise Linux distributions typically have SELinux enabled and enforcing by default. This mandatory access control allows a privileged process, say a web server for example, to be constrained in what it can do. It may be allowed to open port 80 , a privileged operation, but it can still be prevented from modifying the password file. This means that if the web server contains a vulnerability and is compromised the attacker will find it difficult to leverage their control of that privileged process into complete control of the system. The default policy does allow any attacker who successfully gets a root shell to disable SELinux (something we don’t allow on distributions protected with Titanium Linux) but enabling the SELinux LSM significantly reduces the number of paths to a root shell and this is a strong security move.
Secure Software Development, Build Options, and System Configuration: This area has improved drastically. In the bad old days, it wasn't uncommon for distributions to start vulnerable daemons by default. In addition, a number of helpful build options are now in common use:
No-Execute Bit (NX/XD), protecting against running arbitrary code on regions of memory that aren’t marked explicitly for execution.
Address Space Layout Randomization (ASLR), randomizing the memory layout of the OS and applications so that a successful attack on one machine is not likely to affect another.
-fstack-protector-all, a compiler flag that causes the code to include a stack guard variable. Before returning from a function this guard variable is checked to see if an overflow attack has overwritten making these attacks much more difficult.
The Bad
Data at Rest (DAR) Encryption: In embedded systems protecting sensitive system data that lives in long-term non-volatile storage when the system is powered down is critical. These systems are often in untrusted environments and an attacker will always take the opportunity to steal secrets by dumping flash memory while the system is powered down and unable to protect itself. Enterprise Linux systems typically have the ability to enable LUKS encryption, which is the most common method for encrypting storage devices in Linux, however, it is not enabled by default.
Secure Boot: While there are ways to achieve secure boot, they are often complex and/or poorly supported. Given the limits of DAR encryption mentioned above, there is often no clear path to decrypting your data if secure boot has determined the system is in a trusted state. This is a problem our Titanium Secure Boot product aims to solve for embedded Intel systems, protecting not only against modification or subversion of core system software such as the OS or Hypervisor using attacks such as Blue Pill but also further protecting data at rest by preventing decryption unless the system is in a trusted state.
The Ugly
Attack Surface Reduction: This is the single area where enterprise distributions most conflict with the principles of secure software design. The maintainers are continually striving to add more software packages, more kernel-level drivers for broader hardware support, and more capabilities in general. This is good business sense on their part. More features, capabilities, and support means more customers that can be satisfied. However, it is a big problem from a security point of view.
How many of those software packages contain zero vulnerabilities? How well tested and audited are all those kernel modules and drivers? The answers are a) none, and b) not many, leaving many of these packages and drivers vulnerable to attack. The more lines of code available to an attacker, the more opportunity to exploit an unforeseen vulnerability.
Hardware Resource Partitioning: Because these distributions are designed for cloud and data center operations, they are almost always configured to share hardware resources (such as CPUs and memory) with any process that requests them. This can lead to information leakage as attacks like Spectre have demonstrated.
A Note on Old Versions
Enterprise distributions tend to be adopted by large institutions and see use long after their heyday. Despite attempts by the maintainers to apply security patches to this old software, they simply cannot keep up with the evolving threat landscape. Many of the "good" items above aren't available in these old versions while all of the "bad" and "ugly" items are worse. Trying to protect a distribution released more than a decade ago is exponentially more difficult than a modern one.
Our Recommendations
We encourage engineers to consider if what they are building is a cloud/datacenter application or an embedded system when designing their system and choose a Linux distribution accordingly.
If it is a cloud/datacenter solution, then Enterprise-type distributions will likely be a good fit. However, if what needs to be built is an embedded or cyber-physical system, we encourage the use of an embedded distribution, for example Wind River Linux, or even Android. Those looking to take attack surface reduction one step further should consider the meta-distribution tools such as Yocto or Buildroot which allow you to “roll your own” Linux distribution with the exact software packages you need and nothing more. In short, matching your distribution to your problem domain is good engineering practice.
And finally, for your sake and ours, and no matter which distribution you choose, please, please use the newest version possible, and make a habit of updating your images frequently, and follow our 7 Tenets of Layered Security in Embedded Design to make your embedded system as resilient as possible against modern cybersecurity threats.