A Step-by-Step Guide to Defend Against “Bring Your Own Filesystem” (BYOF) Attacks
What is a “Bring Your Own Filesystem” attack?
Bring your own filesystem (BYOF) attacks have become increasingly common. In a BYOF attack, an attacker delivers payloads to a target, as it minimizes their footprint and system-level interactions. From a practical perspective, how would (or could a system designer) defend against these attacks? Even more so, how can a system designer implement proper defenses to even prevent similar types of attacks in the future without just trying to plug all the possible holes retroactively.
Much like containers, a BYOF attack enables an attacker to minimize or eliminate external dependencies, minimize, or evade detection, and gain the ability to execute cross platform. This attack type also enables an attacker to use a filesystem image to contain all their tools, associated libraries and any additional dependencies.
We’ll walk through how a BYOF attack is executed, and at each step identify the defense tactic that can be implemented to prevent and/or mitigate success by the attacker. It goes without saying that BYOF attacks are complex. In order to combat a BYOF attack, system designers can use a variety of capabilities within a Linux system to help defend against these attacks.
BYOF Attack Types
From a practical perspective, BYOF attacks can be grouped into two categories: those that require root or privileged operations and those that can be executed in userspace without privileged operations. Maybe not surprisingly, while many of the possible defenses for both approaches are the same, there are also differences to address. System designers should implement a variety of techniques and defenses to establish defense-in-depth for our system to prevent the introduction of new and novel attack methods, mostly focused on getting user execution on the target.
Most bring your own filesystem style attacks are focused on eliminating external dependencies, minimizing exposure to various analysis capabilities, and have focused on non-privileged operations. Let’s start by looking at an example situation using the style which requires privileged operations, as they expose a few more subtleties that become important when looking at how to defend against unprivileged attacks.
How does a BYOF Attack Happen?
To set the stage, let’s presume there is an attacker with privileged execution of a system, be it an In-Vehicle Infotainment (IVI) system or PLCs running Linux. If we start out with the attacker having a system shell, it’s easier to work thru, but the same concepts apply if it’s the ability to execute arbitrary shell commands or memory-based attacks used to make arbitrary systemcalls. Let’s also presume the attacker has a filesystem image that has been downloaded or otherwise made available on the system. As has usually been the case, this filesystem image is written to a writeable area (for instance, in a user’s home directory or someplace like /tmp). There is nothing that says the filesystem image can’t be memory mapped or even something more exotic like sshfs.
This filesystem would normally be mounted using something like “mount -o loop /tmp/filesystem.img /mnt/myfs”. To execute a successful attack, the attacker needs to be able to do a couple things: call the mount system call, setup a loopback device (in order to mount a local filesystem image as a block device), and then execute new applications / libraries from the new mount point. The attacker may also need to load additional filesystem drivers (i.e. kernel modules), so the system recognizes or supports the specific filesystem image they have brought along. Backing up another step, the attacker also needs to download the filesystem (P.S. This gives the system designer more ways to detect and prevent this type of attack).
Defending a BYOF Attack by a Privileged User
Preventing Image Download
Initially, the system designer can try to thwart the image download through a variety of firewall or local network techniques. Using Netfilter or iptables, we - the system designer - can enable outbound filtering on the device, such that the device can only reach known good hosts (maybe our log collection systems, provisioning systems, and update servers). However, an attacker with privileged access to the system can disable the firewall rules, retag packets, or even attempt to get more exotic and use source routing. While enabling an outbound firewall is a good start, this specific scenario also helps makes the case for defense-in-depth. We need to have defenses on the far edge, as well as the network level. We need to have a higher-level firewall in place to enforce these network rules, off the device itself. Similarly, at the network-level, we could use a variety of application / protocol specific firewalls, explicit route table entries, null routing, and protocol filtering to help ensure our devices only do exactly what we intend and nothing. Similarly, we can also enable both local and remote DNS filters (obviously, this doesn’t work if the attacker downloads the image using an IP address, unless we have a very strict IP allowlist in place). With these defenses in place, it makes it harder for an attacker to download the initial image, although they still have a variety of techniques (such as DNS padding, extra HTTP headers, encoded parameters, sneaker net, etc.) available to them.
Limiting Writeable Locations and Write Permissions
In addition to preventing or mitigating the download of the filesystem image itself, the system designer can also take steps to inhibit where this image can be written or saved and place additional restrictions on these filesystems. These images need to be saved to a writeable filesystem or directory. As a general rule, the system designer should follow the POSIX standard and use separate mount points / filesystems for the various directories, which will ensure a read-only root file system, as well as limit writeable locations and various other security relevant flags. Depending on the system design, the system designer could enforce only locations such as /var/log and /tmp being writeable, as well as limit the size of the directories. Similarly, mounting these writeable locations with mount flags that include: noexec, nosetuid, nodev mitigates a variety of attacks and makes it more difficult to execute and/or create new device entries on these devices. Of course, because the attacker is executing with privileged access to the system, they can remount the filesystems with the flags removed and/or update /etc/fstab to remove the flags on reboot. Protecting against the remount requires restricting system calls, and potentially even other kernel-based capabilities which would be out of band for an attacker. Similarly, we need various protections in place to prevent an attacker from modifying /etc/fstab, various init scripts, the kernel configuration, and potentially even the filesystem meta-data itself.
Demonstrating the defenses required at multiple points in the attack surface, it becomes clear that there is no single solution to address all threats or provide all defenses. It’s critical for system designers to fully understand the Linux environment they are operating within, and leverage what’s there, while working through how an attacker would execute a variety of attacks. As a general-purpose environment and OS, Linux provides multiple avenues of protection, defense, and a myriad of capabilities that all need to be addressed.
Minimizing the Attack Surface
The next defense the system designer can use is narrowing the attack surface with the set of available filesystems in the kernel. Broadly speaking, this can be approached two ways, either by building a monolithic kernel with all the required drivers in it, or by mitigating which loadable modules are present and can be loaded. In most systems, integrators often choose to use a kernel with loadable modules, which presents several additional challenges, namely the need to enable kernel module driver signing and ensure the signing keys aren’t available to an attacker. Otherwise, an attacker can continue with the filesystem attack and bring their own signed driver as well. Identifying which filesystems are needed may not be easy, especially if using the same filesystem images for multiple devices or usecases, as is common in industry. As an example, most of the systems might use ext4 for the rootfs , and perhaps for updates or maintenance tasks the system also has to support msdos / fat filesystems, giving an attacker additional avenues for their own filesystem. Identifying what filesystems are needed is a little more complicated than just looking at your rootfs, as the system designer should also look at what the system itself needs. The underlying system may use filesystems such as procfs, sysfs, devfs, and overlayfs - just to name a few. One often overlooked filesystem that can even be used by unprivileged attackers is fuse, which essentially permits userspace filesystem drivers to be used. Notice that the mount example made use of the loopback filesystem support, which is another candidate for disabling within the kernel depending on other system-level components and interactions. It should be noted, that removing unnecessary or unneeded filesystem drivers does not make it impossible for an attacker to bring their own filesystem, it just makes it harder and maybe less portable, depending on the filesystems we use and what the attacker is using.
One finer point that may be overlooked with the variety of filesystems is where the filesystems are mounted, where they can be mounted, and with what flags they can be mounted. As an example, an attacker with privileged access to the system can choose to remount a filesystem with executable flags, or with SELinux disabled on it, potentially subverting some of secure design principles and access controls. Similarly, and an area we have been focusing on for Kevlar Embedded Security from Star Lab, is an attacker can mount a new filesystem at various sensitive locations, potentially providing another avenue for an attacker to change or influence the system. This is important for services like systemd, which let you dynamically load / exchange service configuration files (therefore disabling other security relevant capabilities we’ll get to shortly), or even something like pam which controls authentication and system configuration tasks. This should help drive home that system designers need defense-in-depth and need to assume that an attacker will get system-level access. From that perspective (or threat model), how can system designers make their systems stronger? How do they prevent an attacker from remounting filesystems, or adding new driver signing keys to the keychain, or disabling security-relevant capabilities?
Restricting System Calls
Using the example from above, an attacker ultimately needs to be able to execute the mount systemcall, either directly or via a variety of in-memory based attacks. A Linux environment provides the ability to restrict system calls for an application through seccomp. In this case, where an attacker has shell access, there are immediately a couple of roadblocks. Shells (i.e. bash, bourne, zsh, etc.) don’t natively support the use of seccomp (or more generally make use of the prctl system call to set up the system call filters), and without a way to preserve file integrity, even from a privileged attacker, the attacker can modify the binary (or potentially play library loader games to bypass or override the filters, but we’ll cover that shortly with allowlisting). Withhout defense in depth emoployed here, then an attacker can subvert the system designer’s attempts to limit systemcalls. Then, there’s the need to contend with the shell being using to start / kick off system services. Many of these system services may require the use of the mount systemcall; therefore, preventing mount from being used by the shell may also hinder proper system operation, making it difficult to enforce or debug.
Related, an attacker who is bringing their own filesystem also is looking to use some form of the chroot or namespace systemcalls. This is a result of their desire to minimize or eliminate external dependencies and essentially execute their payload independent of the rest of the system. So, in addition to just restricting mount and various other related system calls, the system designer should also consider system calls such as chroot. It’s no surprise with around 300+ systemcalls, it is necessary to restrict an order of magnitude (or more) of these.
In theory at least, to restrict systemcalls in a shell, there are a couple additional options which could be explored, though from a practical perspective they are usually discarded. The system designer could choose to maintain their own patchset for selected shell(s). This patchset would then enable / enforce seccomp and enable the system designer to filter the mount system call; however, they are likely prevented from pushing these patches upstream, as the patches likely wouldn’t be accepted by the package / shell maintainers. Not being able to upstream patches makes the system patchesharder for system designers to maintain and increases the burden of package / OS maintenance and upgrades. Alternatively, system designers could try to write their own shell. Note that , system designers should avoid writing their restricted shell in a traditional language such as ‘C’, and instead should use a type / memory safe language such as Rust. This is because writing the restricted shell in Rust helps mitigate a wide variety of in-memory attacks which give the attacker arbitrary code execution and eliminate the purpose behind writing the shell in the first place. Replacing the system shells also presents several other concerns, we have to enable root or administrative login on that shell, we may need to update SELinux labels, and we have to do a fair bit of testing / validation.
Using the mount syscall from an in-memory style attack is a bit more interesting, as the attacker would be operating within the context of a (whether known or not known) process or application. Therefore, depending on from where and how an attacker is executing, they will be executing with the seccomp / syscall filters of the execution context. This may or may not be desirable and might force an attacker to use another technique (i.e., object reuse) in order to change their execution context. The system designer would still, however, be able to enforce systemcall restrictions at least dependent on where / how an attacker is executing in memory.
Applying Discretionary and Mandatory Access Controls
Again, using the same example, the last thing the system designer could consider is the application of both discretionary and mandatory access controls to prevent an attacker from executing mount. Discretionary access controls (the traditional *nix rwx applied at user, group and other/world levels) are an easy first step in cases where the attacker has some sort of direct execution (i.e. not in-memory execution). The downside is that an attacker with privileged execution can change the DACs for mount and various other assorted system utilities such that they can still be used. In some cases, DACs may be enforced at the filesystem level, thereby effectively enforced by the kernel. However, there remains the challenge of preventing a privileged user from potentially modifying them (i.e. where a mechanism is likely necessary to enforce offline integrity for the kernel, the kernel command line, and various system-level configuration files). After exploring DACs, the system designer may want to look at mandatory access controls (MAC), using something like SELinux. MAC-based policies provide the ability to effectively deprivilege the root user. Therefore, with SELinux it is necessary to set up separate contexts for the shells, applications available to user, and system services. System services such as systemd would be able to fork and call mount, whereas the shell would not. This, however, introduces another use case to consider, how can a system designer prevent an attacker from using systemd to handle the mount for them? Is the attacker prevented from modifying or overriding systemd’s configuration files or command-line arguments? Additionally, the SELinux policies could be incorporated into the kernel, such that they are prevented from being changed / modified at runtime and have offline integrity.
Applying Application and Library Allowlisting
One area we haven’t looked at much yet, which incorporates every technique we’ve discussed so thus far, is that of application / library allowlisting. Application / library allowlisting enables system designers to ensure only authenticated and expected applications or libraries can be executed. Taking this a step further, since it may not be overtly obvious from that statement, allowlisting also prevents applications or libraries from within an attacker’s filesystem from being executed or loaded. Therefore, assuming the system designer is implementing allowlisting correctly, and can continue to enforce allowlisting for a root-level user (which isn’t always the case), even if an attacker can get their filesystem mounted, the attacker can’t execute anything from it. Thus, effectively preventing this style of attack from being used.
In terms of mechanisms for implementing allowlisting, there is IMA, fapolicyd (mostly used on enterprise distributions such as RedHat), most of the commercial Enterprise-style allowlisting solutions, and Star Lab’s Kevlar Embedded Security. Star Lab has previously written extensively about allowlisting. Kevlar Embedded Security is one of the few allowlisting solutions that specifically prevents a root-level user from disabling, reconfiguring or bypassing the allowlisting capabilities. With respect to an allowlistlisting solution, system designers are trying to prevent or mitigate various aspects of execution:
Prevent a user from executing their own applications (could be downloaded, could be an attacker provider filesystem, OS update, or a modified system application)
Prevent a user from loading or replacing a library into an existing system process. This prevents a user or attacker from bypassing or modifying calls to prctl / seccomf, adding additional system calls, or potentially modifying the excepted flow of our applications. There are a lot of ways to load, replace, or alter libraries loaded by the path including loader tricks / environment variables (i.e. LD_PRELOAD, LD_LIBRARY_PATH), or modifying the system-wide configuration (file). It should be obvious from this last statement, that in addition to just providing application / library allowlisting, system designers need to provide offline integrity enforcement for system configuration files.
As a couple of examples (and we’ll circle back to these when we discuss bring your own filesystem attacks from an unprivileged user perspective), assuming system designers enforce DACs / MACs, restrict system calls, etc, allowlisting also prevents an attacker from executing their own binary that contains a mount (or any other related) systemcall.
Defending a BYOF Attack by an Unprivileged User
Building on the protections we established above for a privileged attacker, we can now explore how a BYOF attack works for an unprivileged user. As an unprivileged user, the attack flow is subtly different, namely that most attackers have been using something like proot. The proot executable is interesting for a couple of reasons. Namely, it can be downloaded directly from github (or really any other source) for multiple architectures, it’s statically linked so it has no external dependencies, and maybe the most interesting aspect, it uses the ptrace systemcall to mimic mount and chroot, such that privileged access is not required.
Before we dig a bit more into ptrace, lets rehash some of the protections we have already covered that could be used to help a system designer to prevent this kind of attack (and even most other attacks that require a download or execution of a malicious payload and any kind of bring your own tools).
Outbound firewall and application-level firewall: prevent the tools / images from being downloaded. As an exercise to the reader, how would this change if say a hosted version of wordpress was used to download the image / payload, and maybe also perform the execution phases?
DNS and protocol level filtering
Split out filesystems, limit writeable locations and set appropriate filesystem mount flags
DACs / MACs
System call filtering
Application / Library allowlisting
From a user-interactive perspective it may become slightly more viable to provide users with a much more restricted / customer shell. Using something like busybox, the system designer may be able to build a custom, very limited user-facing shell that also enforces seccomp filters for forked / exceed applications. This may work as the user-level shell being used to launch various system services / applications is not a concern, and it also gives the system designer more control over execution path, what can be executed, etc.
Since proot uses the ptrace systemcall in lieu of mount/bind/chroot, the system designer should add ptrace to the systemcall filters that are enforced / removed from a user. The ptrace systemcall is also used by application debuggers such as gdb, and even by applications attempting to do their own address resolution for libraries and/or other dynamic content.
The most important protection to provide here is application / library allowlisting. First and foremost, this prevents an attacker from bringing their own tools, whether it’s a helper application like proot, their entire filesystem, or whatever malicious payload the attacker desires. The use of application allowlisting here is effectively the first line defense against someone compromising the system, as they are forced to use only what is on the system already. Then, the system designer gets to backstop the allowlisting approach with all the functionality discussed above, so that the system dessigner has defense-in-depth, and can prevent a wide variety of attacks from being successful. As we can see, a proper implementation of application / library allowlisting, backstopped with a variety of other defenses, including offline integrity enforcement for system files (i.e. applications, libraries, and configuration files), provides a solid foundation to continue building the security environment. Allowlisting also helps prevent many attacks and attack paths - not just BYOF attacks - and really does establish the foundations of a secure computing base.
Hopefully, you now have a better understanding of how complex attacks such as BYOF attacks work, and how system designers can use a variety of capabilities within a Linux system to help defend against these attacks. In fact, these defenses work for any attack involving an “off platform” payload. No one standalone defense tactic is sufficient to prevent these attacks. To effectively defend the system, system designers must consider multiple layers of the threat model and defenses at each of those layers. As a general-purpose OS, Linux security is extremely complicated, and there is no single toggle or application package that can be applied that addresses all various security challenges for embedded systems.
At Star Lab, our Linux security experts, with decades of industry experience creating solutions for critical systems, have designed a security solution for embedded systems that takes care of the most cumbersome elements of security implementation for you. Reach out to us for help with a security assessment or to learn how Kevlar Embedded Security can help address your security challenges, saving your team time and money building your own security security solution.