There is no silver bullet to security, and even more importantly, there is no single source of truth for what security options are available, what they do, and what impacts they have, or even how they all work together.

All too often, it is assumed you can apply a STIG (or similar configuration guidance) and magically produce a secure system. The available STIGs assume a one-size-fits-all solution without considering the specifics of a single purpose embedded system. The STIGs may provide a foundation for starting down a secure path (for a given threat model), but they don't begin to address the possible security configurations or tailoring that may be required for operational environments.

In this post, we will enumerate the many security mechanisms that can be operationally deployed in an embedded system running Linux in order to secure that system from common software attacks. While the traditional approach to security relies on the CIA triad of confidentiality, integrity, and availability, the majority of this post is primarily focused on the principles and enforcement of integrity.

For many deployed systems, they are using Linux distributions with a slightly older kernel, which potentially limits the use of some newer, security relevant features which have been introduced into later kernel versions. Some of these newer kernel features, such as LSM stacking, may complicate the decision-making tree in a fashion that isn't representative of deployed systems as they introduce an additional layer of configuration and integration.

Threat Scenarios

As we evaluate different security configurations, we'll generally look at the configurations with respect to the following scenarios:

Network-based protections (i.e., keep an attacker out)

Application Protections
Multi-user Environments
Execution Pivot
Traditional Information Assurance (IA) protections

Using these threat scenarios, we can deduce many reasons for why defense-in-depth is required, and similarly we can see where configuration guides such as a STIG fall short. Case in point, if a user is able to bypass network / application style defenses and gain arbitrary execution on the system, they are then in a situation where they will be working to elevate their privileges, in order to carry out the rest of their attack. It is this scenario that we are primarily concerned with when we say an application can only access these

files, sockets, and IPCs and nothing else. We are focusing on preventing an attacker who compromises an application from pivoting, elevating permissions, and establishing a foothold on the system.

The Foundations

Before we move into various kernel and OS-level security configuration options, we need to stress the importance of secure boot. At a bare minimum, whatever secure boot option used has to enforce integrity of the kernel, kernel command line, and initramfs. The kernel command line and boot arguments enable an attacker to alter system behavior, mitigate or bypass various security relevant features, and change various configuration options at runtime.

Which specific secure boot solution is used is a function of available hardware, the operational environment, and any specific constraints of the system.

Generally speaking, secure boot options include:

UEFI Secure Boot
Trusted Grub
tboot or Titanium Secure Boot (using Intel's TXT extensions)
Password protected Grub
Vendor specific solutions

Kernel Configuration

These changes will require the kernel to be recompiled, although there are other reasons the kernel will need to be compiled as well, including attack surface reduction, removing unused drivers, performance tuning, etc. The kernel that ships with many popular Linux distributions is intended to be a one-size-fits-many and should be tailored for specific instances and use cases.

This is only a brief look at various security-relevant kernel options and is not inclusive. Depending on the target system, use cases and environment, it may not be possible to alter some of these settings as a result of legacy software, performance, or other operational constraints. Additionally, many of these configurations are platform dependent, and we are only exploring x86_64 configurations for brevity.

Kernel Configuration Options

CHECKPOINT_RESTORE - Enables specific code and memory execution on various crashes / failures (could lead to arbitrary code execution). [Security recommendation: CHECKPOINT_RESTORE=n]

BPF_JIT_ALWAYS_ON - Enables the BPF subsystem to dynamically (executable) code into the running kernel (Could lead to arbitrary code execution and subverting security features) [Security recommendation: BPF_JIT_ALWAYS_ON=n]

USERFAULTFD - Enables page faults to be handled in user space and could be used to access memory that shouldn't be accessible or could lead to information leakage. [Security recommendation: USERFAULTF=n]

COMPAT_BRK - Disables kernel heap randomization, potentially making it easier to exploit various memory overflows in the kernel. [Security recommendation: USERFAULTFD=n]

SLAB_FREELIST_RANDOM - Enables or disables the randomization of the kernel's heap, potentially making it easier to exploit kernel heap overflows. [Security recommendation: SLAB_FREELIST_RANDOM=y]

SLAB_FREELIST_HARDENED - Protects the kernel slab's metadata, potentially making it harder to execute various slab / heap attacks. [Security recommendation: SLAB_FREELIST_HARDENED=y]

CONFIG_SHUFFLE_PAGE_ALLOCATOR - Increases the randomness of page allocation addresses. [Security recommendation: CONFIG_SHUFFLE_PAGE_ALLOCATOR=y]

Kprobes / Oprofile / GCov - Enables tracing the kernel, and filesystems, potentially leading to information disclosure and facilitating certain reverse engineering tasks. On a deployed system, these should both be disabled and removed from the kernel.

STACKPROTECTOR - Enables GCC's stack protector for kernel memory, making it harder to exploit various kernel stack vulnerabilities. [Security recommendation: STACKPROTECTOR=y]

STACKPROTECTOR_STRONG - Enables stack canaries (under specific conditions) making it harder to exploit various kernel stack vulnerabilities. [Security recommendation: STACKPROTECTOR_STRONG=y]

VMAP_STACK - Enables virtual stack mappings with explicit guard pages, making it more difficult to execute various kernel stack vulnerabilities. [Security recommendation: VMAP_STACK=y]

REFCOUNT_FULL - Forces a full validation of various reference counts on condition, potentially preventing scenarios which could lead to a use after free vulnerability. [Security recommendation: REFCOUNT_FULL=y]

MODULE_SIG - Enables the verification of signatures on loadable kernel modules. Note, enabling this kernel option will force various other constraints on the integrator / deployed system, and requires consideration to the protection of signing keys, enrollment of the (public) keys into the system keychain, and may not be fully supported or easy to use with some 3rd party drivers. Module driver signing explicitly removes one possible attack vector an attacker can use to pivot and/or gain higher privilege access on the system. The security recommendations here explicitly require 2 kernel options, such that driver signing is enabled and enforced. [Security recommendation: MODULE_SIG=y && MODULE_SIG_FORCE=y]

CONFIG_RETPOLINE - Disables the use of speculative execution paths within the kernel (i.e., prevents a variety of side channel attacks on the system). [Security recommendation: RETPOLINE=y]

CONFIG_X86_INTEL_MPX - Enables Intel's Memory Protection Extensions, which enable shadow pages, stack guard, and can be used to prevent various levels of application-level vulnerabilities. Note: MPX extensions require fairly new Intel hardware / chipsets. [Security recommendation: X86_INTEL_MPX=y]

CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS - Enables memory tagging in a virtualization or multiple execution context environment and enables the IOMMU to provide application-specific memory boundary enforcement and optimization. [Security recommendation: X86_INTEL_MEMORY_PROTECTION_KEYS=y]

Intel's TSX - Uses Intel extensions to optimize locking and various operations, however the TSX extensions have also been shown to introduce various side channel attacks into the system. [Security recommendation: X86_INTEL_TSX_MODE_ON=n]

CONFIG_KEXEC - Enables a new kernel or ELF image to be loaded without reinitializing hardware. KEXEC has the potential to bypass secure boot, and expose secrets resident in memory. [Security recommendation: KEXEC=n && KEXEC_FILE=n && KEXEC_JUMP=n]

CONFIG_RANDOMIZE_BASE - Enforces address space layout randomization for the kernel, inhibiting a broad variety of potential attacks against the kernel and data structures. [Security recommendation: RANDOMIZE_BASE=y && RANDOMIZE_MEMORY=y]

CONFIG_LIVEPATCH - Enables the kernel to be dynamically patched at runtime, exposing multiple potential attack vectors and potentially enabling an attacker to disable various security protections. [Security recommendation: LIVEPATCH=n]

CONFIG_FUSE_FS - Enables virtual, user space filesystems to be created, potentially enabling security features to be bypassed and/or exposing various security vulnerabilities. [Security recommendation: FUSE_FS=n]

Kernel Hacking - The kernel hacking subsystem enables a variety of features that provides developers access and insight into the kernel, potentially bypassing ASLR enforcement, enabling debugging, etc. [Security recommendation: All options under this submenu should be disabled]

CONFIG_SECURITY - Enables the kernel to provide multiple facilities for security modules and configurations, potentially enabling an attacker bypass security enforcements. Note: This will also require you to configure an explicit security module such as SELinux. [Security recommendation: SECURITY=y; Just note setting this to Y will require additional configuration and it’s not inclusive.]

CONFIG_PAGE_TABLE_ISOLATION - Unmaps kernel address space from user space and helps prevent various hardware-level side attacks. [Security recommendation: PAGE_TABLE_ISOLATION=y]

CONFIG_HARDENED_USERCOPY - Perform additional verification on the copying of data from user -> kernel space (i.e., as part of a system call), making it more difficult to conduct a variety of errors and vulnerabilities. [Security recommendation: HARDENED_USERCOPY=y]

FORTIFY_SOURCE - Forces the compiler to verify string copies when building the kernel; can help prevent a variety of traditional buffer overflow errors within the kernel. [Security recommendation: FORTIFY_SOURCE=y]

STATIC_USERMODEHELPER - Forces the kernel to use a single user mode helper, which dispatches the correct helper. User mode helpers are used to process binary formats, mount filesystems, etc., and enforcing restrictions around user mode helpers makes it harder to subvert protections or execute a variety of privileged escalation attacks. [Security recommendation: STATIC_USERMODEHELPER=y]

CONFIG_LOCK_DOWN_KERNEL - Disables various kernel interfaces that could be used to execute a variety of side channel and privilege escalation attacks. [Security recommendation: LOCK_DOWN_KERNEL=y && LOCK_DOWN_MANDATORY=y]

SECURITY_SELINUX - Enables the SELinux Mandatory Access Control framework, and various support options. As we've previously discussed in our blogs, SELinux enables MAC-based policy to be applied to applications, data and users. Additionally, SELinux supports the tagging of network traffic (i.e., in the case of virtualized or docker environment, where you wish to restrict specific traffic to specific execution environments). Note: The recommendations below assume you have a valid and fully functional policy for the system, as they will prevent the policy from being developed on the running system. This is a great example of where a crawl, walk, run approach is needed and all of the pieces will need to come together at the end. [Security recommendation: SECURITY_SELINUX=y && SECURITY_SELINUX_BOOTPARAM=n && SECURITY_SELINUX_DISABLE=n && SECURITY_SELINUX_DEVELOP=n]

CONFIG_SECURITY_LOADPIN - Forces the kernel to only load firmware, kexec images (if enabled), security policy (i.e., for SELinux), modules, etc. from the root (or other statically defined filesystem). Inhibits various attack vectors and rebinding of filesystem mount points. [Security Recommendation: SECURITY_LOADPIN=y]

CONFIG_INTEGRITY - Enables the integrity measurement subsystem, which provides a possible solution for addressing application whitelisting. IMA provides an alternative to Titanium Technology Protection's whitelisting/allowlisting and forces various operational and performance constraints on the system, and if Titanium Technology Protection were used, you would likely want IMA disabled. [Security Recommendation: INTEGRITY=y && IMA=y && IMA_APPRAISE=y && CONFIG_EVM=y]

Additional Kernel Details

There are, of course, numerous other security relevant kernel options and configuration details that are highly dependent on specific use cases, specific networking needs, and targeted applications. In just the networking section of the kernel configuration, there are literally hundreds of security relevant configurations, which need only be selected if they are required for a specific use case. As an example, there are options to enable extended attributes and tagging of network traffic, which would be most useful in an environment with multiple containers or virtualization environments where it's desirable to separate and isolate network traffic to specific execution domains. Similarly, there numerous security relevant options (i.e., labeling, extended attributes, etc.) that are only relevant or applicable to specific filesystems which may be used by the platform.

Many of the specified security options require additional details (i.e., specifying the user mode helper) during kernel configuration. Similarly, many of the specified security options require an incremental approach and very likely separate development and provisioning systems in order to tune specific applications and or settings. We didn't specifically call it out above, but some of the options specifically enforce operational constraints which may conflict with how the systems have historically been used. While it is desirable to enable all of the options as identified above, the operational and/or performance constraints may significantly hinder the ability to enable some options. This is especially true for legacy systems undergoing a tech refresh, in which case as many of the options as possible should be enabled.

Connecting the Dots

Sticking with our threat vector, the idea is to constrain what an application can do and prevent an attacker from being able to pivot to either a higher privilege level (i.e., user privileges, or user->kernel privileges), or modify the resulting system, there are numerous other Linux capabilities that should be used. Leveraging these capabilities and configurations will help enforce defense in depth and complement the kernel options above. It should be noted that the configuration options identified below are only a representative set, and should not be considered inclusive, and they may require additional mitigations to protect against newly introduced threat vectors.

Combining enough pieces, we can work towards not having a highly privileged user on a deployed system and make it very difficult to pivot and/or gain a foothold on a system. In essence, the defense-in-depth approach enables us to drive towards "single purpose" Linux environments.

Discretionary Access Control (DAC)

Linux, and most other POSIX operating environments provide coarse DACs which often consist of user, group, and world permissions. DACs establish the building blocks which can be used to prevent access to specific directories or applications, but they are also limiting.

In most POSIX environments, you can apply DAC policies to only a single group. With traditional DAC enforcement, you cannot often enforce constraints such as:

The webserver can only access these specific files (and nothing else). In order to help combat the limitations imposed by the traditional DAC mechanisms, many filesystems now support the use of extended attributes.

Extended attributes enable fine grained permissions, multiple group-level assignments, and pave the way for mandatory access controls, and whitelisting techniques such as IMA. Additionally, extended attributes can intern be verified (so you can verify they haven't been altered) by IMA and similar approaches. However, extended attributes alone pose another problem and generally assume a highly privileged administrator (thereby encouraging an attacker to attempt to elevate privileges and become the "super user"). This inherent weakness leads us down the path of the following sections and integrations, such that we can achieve defense in depth.

Mandatory Access Control (MAC)

MAC engines such as SELinux get us closer to being able to enforce constraints such as an application can only access these specific files, sockets, IPCs and nothing else. However, MAC engines also assume that there is a highly privileged user (i.e. "root") on the system, thereby enabling an attacker a mechanism to attempt to gain root-level access.

In the case of SELinux, a targeted policy should be used and all objects on the system should be labeled. The targeted policy should be tailored for a specific application or use case, such as providing appropriate labels and policies for a user defined application. Additionally, if the kernel is configured correctly as identified above, the SELinux policy cannot be changed, and the effects of these various forms of pivot or attempts to become the “super user” will fail.

Capabilities

Linux capabilities provide a mechanism to enable an unprivileged process (i.e, one restricted using DAC or MAC) to perform certain privileged operations. As an example, this enablement (or restriction) enables applications to bind to a low port while still running as a normal user. This can make it harder to pivot or exert full control over a system as the initial access vector or compromised application may have been limited in how it can interact with the rest of the system.

Capabilities can be set specifically by the application (i.e., through a system call), which is commonly used by setuid applications or through the application of extended attributes. Additionally, capabilities can be restricted through the use of cgroups and namespaces (the same building blocks docker and other container engines use to isolate their applications). In the case of using extended attributes to enforce application capabilities, it will also be desirable to use IMA or a similar capability to enforce the integrity of the extended attributes and the permissions.

The Linux capabilities man page provides a full list of all capabilities that can be set and established for a process or application.

Capabilities should be applied to all applications and services that do not need full "super user" privileges to perform their operations.

systemd in All its Infinite Glory

Most Linux-based systems have transitioned to the use of systemd for service and application management and system-level configuration. Systemd might as well be a complete OS given its complexity and the variety of ways it can be configured and used. Recently, organizations such as RedHat have been expanding systemd’s capabilities, and even documenting some of the more advanced (but, equally important) security-relevant configurations (https://www.redhat.com/sysadmin/mastering-systemd).

Sandboxing

Systemd's sandboxing integrates multiple functional components and/or features and works to restrict how applications can be used. Sandboxing gets us even closer to the scenario of this application can only access these files and nothing else and removes many of the threat vectors as identified above. Sandboxing uses names and cgroups to enforce capabilities, extended attributes, MAC, and various other security options. Sandboxing should be used for all applications and services managed by systemd.

chroots

systemd provides a native chroot interface which builds on sandboxing and even further enforces the use case of *this application can only access these files and nothing else*. In this scenario a new construct is introduced, such that all files including libraries and configuration files needed by a specific application are relocated (either physically or through the use of hard links) into a new root directory, and the `chroot` system call is used to set the new

root for the application such that it can only access the files in its own chroot. It should be obvious, but within a chroot all of the other constraints and components including DAC and capabilities are enforced.

Where practical, chroots should be used for all system-level services, and public facing services such as a web server.

System Call Restrictions

Systemd also provides a mechanism to remove system calls from an applicator service. This is very similar to the seccomp-bpf approach described below, and in effect uses the same kernel interface behind the scenese, as it uses seccomp profiles. Additional details on using systemd’s seccomp filters can be found in the systemd.exec man page.

Whitelisting

Depending on the underlying implementation, (application) whitelisting provides a strong defense against an attacker bringing their own tools (or applications) and being able to achieve persistence and/or arbitrary code execution. Without being able to introduce new tools, an attacker is forced to work with what they have available on the system. Whitelisting is an important consideration to limit an attacker and help defend against pivots and privileged escalation.

Implied, but not explicitly stated, a strong whitelisting solution should provide an integrity mechanism, which is cryptographically verified and enforced, and which introduces its own constraints (keeping the keys out of an attacker purview, system updates, supported filesystems, etc.).

Restricting System Calls

Similar to how Linux supports removing specific capabilities from an application, it also supports removing specific system calls from an application.

The seccomp-bpf framework uses BPF (which if you're astute, you'll notice that we partially disabled above; we specifically disable the just-in-time BPF interface, which is used for dynamic tracing and can be used for implementing interpreted LSMs, while leaving the core of

BPF enabled, so we can use it to filter system calls) to limit the system calls that can be used by an application. On x86_64, Linux currently has 313 system calls defined, and it's almost certain than a single application only uses several dozens of these. The use of seccomp-bpf enables the remaining system calls to be removed from an application, thereby reducing attack surface and greatly inhibiting how an attacker can gain access, pivot, and elevate privileges. One you start using BPF filters for an application, you also need a mechanism to enforce the integrity of the BPF filters for the specific application.

Restricted system calls through seccomp-bpf are commonly used by systemd, docker, qemu, and Chrome. Additionally, they should be applied to all applications on the system.

Wrapping Up

As should be evident, there is no silver bullet for security. You cannot simply toggle a switch, and magically have a secure system. Similarly, there is no one size fits all solution, and each solution or toggle must be carefully evaluated with respect to each system and application being developed. Security needs to be firmly entrenched into system development activities and evaluated with respect to specific threat vectors. It must also account for how likely given scenarios may be in an environment.

Defense-in-depth is fundamental to Linux security. As we’ve explored above, often enabling a specific security option, requires additional configuration. This additional configuration may require the selection of additional options, or the specification of a static configuration element. Some security options snowball, and also require additional security options to be selected in order to mitigate new threats that were introduced by the first security option. As an example, we force SELinux configuration and prevent both on system policy development, as well as disabling SELinux at runtime. Clearly, there is an order of operations here that must be followed and the collection of security options forces specific operational best practices to be exercised. Mitigations need to be layered and must work together to address a variety of threats to the system.

For further reading, the options above are the building blocks that go into our "7 Tenets of Layered Security in Embedded Design”

The Linux Security Hardening Checklist for Embedded Systems