How to Protect Embedded Linux with Star Lab’s Kevlar Embedded Security

07/08/22 Update: Version 1.1 has officially launched! Updates include Limited ARM64 support, and updates to File System Integrity and audit logging. See the details here.

Embedded device security is an overwhelming task. Either there are innumerable requirements, no requirements, or you are tasked with writing your own internal requirements.

At the end of the day, even if you’ve addressed all of these requirements, will that have been adequate to keep your device secure? Maybe not. That’s because your embedded device just might end up in the hands of an attacker...quite literally. Therefore, ensuring your software and data is safe when (not if) an attacker breaks in is essential. This is why, while it’s tempting to jump into identifying security solutions that “check the box” for your requirements, it’s often more helpful to start by first taking a step back.

Defining the Threat Model

We always recommend starting by defining the threat model for the system (as we’ll discuss more in a future blog), or more aptly, defining what types of threats are a concern for our system. More so, we spend time identifying what needs protection and how much protection is needed.

Defining a threat model doesn’t have to be complicated either. We can simplify the process by answering questions like:

Does an attacker have physical access?
Are we concerned with logical or over-the-wire attackers?
How do we handle fielded updates?
What kind of storage does the device have? Is it read-only?
Is the system updated regularly, or is it expected to be static over its lifetime?
Do we have performance restrictions?
Do we have specific security requirements?

If we look at various security requirements such as FIPS 140-2 or FIPS 140-3, we see a similar approach to threat modeling. Within the FIPS, there are four levels of protection that increase in complexity from simple integrity checks and procedures to a model that assumes the attacker has hands-on physical access to the system and will use any means available to gain access. These levels are commensurate with the system, the data it protects, and the expected threats to the system.

Similarly, asking questions about what we’re trying to protect allows us to identify necessary protections:

Are we trying to protect (i.e., enforce confidentiality for) provisioning keys or certificates?
Do we want the system to only operate as we designed and tested?
Do we care about network traffic?
Do we need to introspect data flows between applications?
Do we have driver signing keys?

Unless you understand the threats to protect against and where that data resides in the system (both at rest and at runtime), you won’t know if you need a solution like Star Lab’s Kevlar Embedded Security in the first place; though enabling a baseline security posture with a tool like this helps address a wide variety of threats regardless of your threat model. Here’s how.

What doesn’t Star Lab’s Kevlar Embedded Security do?

Just as understanding what Star Lab’s Kevlar Embedded Security does is important, it’s also important to understand what it does not do, so you can identify what other defenses or protections you need for your system.

Star Lab’s Kevlar Embedded Security:

Does not enforce restrictions on any kind of network activity including ports, sockets, and applications originating network traffic.
Does not check or manipulate containers.
Does not introspect or monitor data flows.
Does not check for logic or programming errors

It’s probably worth repeating again, but Star Lab’s Kevlar Embedded Security is not a one size fits all solution to address all security challenges. It enables defense in depth and provides fundamental security blocks required for a secure system.

What does Star Lab’s Kevlar Embedded Security do?

During the design phase for Star Lab’s Kevlar Embedded Security, we made several assumptions:

It must provide protection even when an attacker has root-level access (an approach that continues to be proven critical when new CVEs such as the POLKIT vulnerability are identified)
It must provide a flexible, layered solution where an integrator or system developer can choose the specific protections they need to satisfy their threat model and protection requirements
It will not likely be the end-all be-all security solution and will likely be combined with other solutions

Star Lab’s Kevlar Embedded Security helps establish a cybersecurity baseline for an embedded system during the system build process and manufacturing. It is available as an add-on to Wind River Linux (WRL) LTS 21 and similar yocto-based distributions. The product release includes additional layers for WRL that are added to your (build) environment via bitbake, devshell and the inclusion of feature flags.

In addition to the base layers, the product provides a variety of helper classes and wizards that facilitate the use of various aspects of it. Out of the box, the default configuration and use of all pillars should work for most systems; however, the individual layers or features can be tailored to address specific use cases as required.

As such, Star Lab’s Kevlar Embedded Security presents the user or developer with an “easy button” to solve specific security requirements and remove many of the design considerations from the user or integrator. As an example, for module driver signing, we need to choose which algorithm to use, how to protect the signing keys, and whether a secure boot solution is necessary. The hardest things to master with respect to security and all the capabilities provided by Linux are how everything works together and what the impacts are for selecting one feature or another.

Similar examples exist with allowlisting. There are at least 3 different ways to do this in a standard Linux environment, all with their own use cases, requirements, and tradeoffs. Star Lab’s Kevlar Embedded Security also aids in solving the problems of poor documentation for the Linux kernel and Linux userspace components (e.g. systemd), and provides an easy path for integration of multiple security capabilities that provide complementary protections.

Version 1.0 is provided as a set of Yocto-layers for Wind River Linux LTS 21 and other Yocto-based distributions. Star Lab’s Kevlar Embedded Security is primarily supported and tested on x86_64 (and the relevant BSPs), however most of the capabilities are generally platform and/or architecture agnostic. There are, however, some caveats that we will be addressing in upcoming quarters and upstreaming the results. Similarly, as customer demands dictate, we will evaluate making some of these capabilities available within other distros such as RHEL or Ubuntu. In this release, you get full documentation and our security-selfcheck layer. The security-selfcheck layer is the same toolsuite we use in our CI/CD environment for testing, which enables you to verify that various features are operating as intended with your applications and environment.

Version 1.0 provides three distinct capabilities that can be used independently or together to address a variety of threats against the system.

Allowlisting – Both online and offline protections (that can be used independently)
System Call Filtering
Enhanced Kernel Hardening

Allowlisting

Allowlisting in Star Lab’s Kevlar Embedded Security is implemented as both dm-verity for offline integrity verification, as well as an eBPF LSM for runtime verification and enforcement. The eBPF LSM forces a read-only root filesystem (or otherwise configured filesystem), ignoring mount flags, continuing to enforce the filesystem restrictions. Both the dm-verity and eBPF LSM components are independently selectable and configured within the Yocto build environment for WRL. Some threat models (i.e., building a backend, cloud hosted system) may dictate only needing one of the components, however if you’re building and deploying a physical asset, you’ll likely need both components and this will be reflected in your threat model assessment.

Why do we need allowlisting with both online and offline protections? Using both protections ensures the continued operation of the deployed system and prevents the introduction of malware, ransomware, droppers, and other malicious payloads through all phases of system operation. Allowlisting verifies that any application or library mapped in for ‘execute’, must have come from an authenticated, or at least a known, source. The use of allowlisting also removes numerous paths for malicious persistence, both online and offline (i.e. while the system is powered off). The use of dm-verity integrated with an existing Secure Boot solution also removes various vectors that can be used to modify, reuse or repurpose a system when an attacker has physical access to the device. One such attack, might be to modify the systemd system call filters for system applications to facilitate another attack. Star Lab’s Kevlar Embedded Security provides two levels of defense against this, preventing modification of the systemd unit file while the system is powered off as well as “live” on a running system.

System Call Filtering

System call (syscall) filtering in Star Lab’s Kevlar Embedded Security is split into two distinct steps: analyzing an application and its dependencies during the construction of the root filesystem and enforcing the system call filters at runtime. At runtime, the syscall filters are enforced using seccomp-bpf filters generated from the pre-processing step. During the analysis of an application and its dependencies, the product builds up a list of system calls used by the application and its dependencies to construct a seccomp-bpf filter list that is added to the application’s systemd unit files. The seccomp-bpf filters are applied to an application by the kernel during the processing of do_exec. The seccomp subsystem of the kernel enforces a 1-way transition into a secure state, meaning that once syscalls have been filtered for a given processes’ address space, additional syscalls cannot be added to the address space. The initial release focuses on applying syscalls filters to applications which are started by systemd, since it provides native support for loading a seccomp-bpf filter and executing the seccomp() system call during daemonization.

Similarly, Star Lab’s Kevlar Embedded Security currently includes proofs of concept for using seccomp-bpf with SysV style startup scripts. Why do we need system call filtering in the first place? If we look at a Linux environment in general, there are around 300 system calls available depending on the architecture and various other configurations. However, most applications only use several dozens of these system calls. While reducing attack surface is always beneficial, what we’re really going after here is making it harder for an attacker to perform various privilege escalation attacks. Several recent CVEs have been issued for systemcalls that can be used to elevate privileges to ‘root’ and these system calls aren’t used by most applications. By removing unused system calls from applications, not only do we decrease the attack surface, but we also make it harder to execute various attacks on the system.

You might be wondering if system call filtering on its own is enough. Let's walk through an example of how allowlisting in conjunction with system call filtering provides defense in depth. The system call filters are read from the systemd unit file which is stored on disk. Allowlisting enables us to prevent the unit files from being changed both at runtime and at rest or offline, thereby helping us to protect the system call filters – even if an attacker has root-level access or can modify them offline.

The real value of syscall filtering is integrated analysis of applications, extraction of the system calls, and creation of the seccomp filters for systemd.

Enhanced Kernel Hardening

The final layer of Star Lab’s Kevlar Embedded Security is enhanced kernel hardening, which enables or disables various security relevant settings in the kernel. The product uses the Linux Kernel Hardening project as a guide for enabling various security-relevant capabilities. These capabilities include enabling and enforcing the lockdown LSM, disabling debugfs, and enabling module driver signing. Additionally, enhanced kernel hardening enables additional verification checks for memory allocations, system calls, and peripheral configuration. Collectively these capabilities make it harder to elevate privileges (especially into kernel mode), mitigate hardware being coopted or reconfigured, and help prevent a variety of information disclosures that could be used to conduct or enable a variety of additional attacks. The kernel hardening piece complements the other layers to enable defense in depth.

We’re excited for the first release of Star Lab’s Kevlar Embedded Security, a solution that fills a critical security need for embedded devices before an attacker breaches the system. The security it provides, and the ease of integration, will alleviate security stress during software development. Future releases will continue to add defense in depth, and of course, enable you to make security relevant decisions. To start a conversation about how Star Lab’s Kevlar Embedded Security can secure your embedded system, click the button below.

Let's Chat