It's not fair.

When attacking an embedded system, it takes only one vulnerability to lead to an exploit, or at least an exploit chain. Of course, this all depends on what the attacker’s goals are in the first place.

This means, when tasked with securing an embedded system, the defender must think through and be prepared to protect against every possible vulnerability, across all layers of the system and overall architecture. Overlook just one opening and the attacker may find it, take control, steal your secrets, and create an exploit for others to use anytime, anywhere.

Worse yet, that same attacker may use an initial compromised device to pivot from one exploited subsystem to another, causing further damage to your network, mission, and reputation.

In this blog post, we present the 7 Tenets of Layered Security in Embedded Design – insight that is based upon decades of experience engineering security solutions across a number of high-value platforms, devices and systems. To be clear, this article is not intended to provide a comprehensive list of all known vulnerabilities. Nor is it a list of every defensive solution that could be employed. Such a list would be impractical for an enjoyable blog post.

Instead, we hope to show you the most important tenets of layered, secure design and best practices that, if adhered to, give you (the good guys) a fighting chance against any attacker who seeks to gain unauthorized access, reverse engineer, steal sensitive information, or otherwise attack your embedded system. One thing to keep in mind, especially with respect to security, there is no single solution that can solve all your security challenges. In order to stand a reasonable chance of defending our systems, we need to employ defense in depth (so no single solution is trying to block an attacker) and practice the discipline of security engineering.

The beauty of these 7 tenets is that they can be layered together into a cohesive set of countermeasures that achieve a multiplicative effect, making device exploitation significantly difficult and costly for the attacker. Not only can these be layered together, but they should be layered together. Multiple approaches to some of the areas may be required depending on the overall system. The individual approaches are also iterative, and they can be applied to your edge devices, your cloud analytics platform, and your software development environment.

In line with the discipline of proper security engineering, and not just treating security as a bolt-on point solution, we need to start our defenses with a proper threat model.

Threat Modeling

At Star Lab, we are huge advocates for threat modeling. We feel it’s so important, we’ve done an entire 2-part series diving deeper into the process and how to do it. Applying proper security engineering to our systems enables us to establish a security posture for our systems and ensure we’re enabling the right security protections. A threat model guides us in selecting appropriate system configurations and options, to determine if we have enough security in place, and to determine if that security sufficiently protects the system from actors with the given level of access.

As part of our threat model, we define our device (or system), where it’s located, how it’s used, and what kind of software/firmware are running. We also need to implement different, or at least additional, security measures if we’re making a physical device, expect it to be physically accessible by an attacker, and realize that a variety of physical reverse engineering and analysis attacks are likely to be employed against the device. On the other end of the spectrum, we have cloud hosted applications in an enterprise or 3rd party data center. As part of our threat model, we also need to consider whether we need to meet specific security requirements or standards, if we need to enforce the confidentiality for specific applications / data, and how we’re going to handle fielded updates.

Data at Rest Protection

Your applications, configurations, and data aren't safe if they're not protected at rest. Period. Depending on your threat model, data-at-rest is either focused on integrity (i.e., making sure it hasn’t been modified), or confidentiality (i.e., preventing it from being understood, viewed, etc.). With most threat models, we’re primarily concerned with the integrity of applications and data. Encrypting our applications or data at rest, may or may not be required. As part of our threat model, we’ll determine if we have sensitive applications / algorithms or secrets such as provisioning or TLS keys that need to be encrypted and protected from unauthorized disclosure.

Integrity for data (including applications) can be accomplished in a variety of ways, and can span the gambit from cryptographic hashes, checksums, redundancy, or even more basic file sizes and other meta data. The level of integrity verification we need is a factor of our threat model. Just because you’re using a read-only rootfs or flash doesn’t inherently mean you have data integrity. As a simple example, we can look at NAND Flash. NAND Flash is inherently unstable and requires the use of ECC bits for error correction as data as read. So even though we’re providing a read-only filesystem to our OS and runtime environment, we still have to consider data integrity from the media we read that data from.

In terms of implementing data integrity, we have a variety of options available to us, especially within a Linux environment. We can cryptographically verify whole block devices (e.g., Dm-verity in Embedded Device Security), validate individual files using something like IMA, or use redundant flash or filesystems. As we found out with our threat model, it really depends on any performance requirements we have, just how secure we can make it, and whether we have a way to protect something like a roothash or set of extended attributes. It’s maybe obvious from that last statement, but we almost certainly want to have the root integrity measurement (assuming we’re building some kind of a tree), bound using our secure boot.

Say your threat model identified that you need to prevent an attacker from observing your application or data, what then?

Sure, if you open up a binary data file or application in a text editor, it looks like jibberish. But with the right tools and to the trained eye you might as well give your attacker the source code.

You can protect your applications and data at rest in one of two ways:

Prevent the attacker from ever gaining access to this information in the first place
Make it impossible for the information to be understood at all

Of course, an embedded system usually has a purpose, and hiding it in a bank vault where it can never be used is not overly useful. Embedded devices are now distributed by the millions to consumers around the world. Medical devices do no good if they don't make it to hospitals for use on patients. Aerospace and defense systems cannot protect and defend our freedoms without being deployed to the field. City infrastructure, buildings, and goods can’t be built without industrial equipment.

All of this is to say, unless you can guarantee that your system remains physically inaccessible behind guns, gates, and (trusted) guards, preventing the attacker from ever gaining access to your data and intellectual property is an exceedingly tall order. This leaves us with method two: Making it impossible to understand the information at all.

Though there are many ways to obfuscate or otherwise garble your data and applications to make them more difficult to understand, most aren't worth the effort and are often trivially bypassed or subverted. When an attacker has access to your software or data, it's only a matter of time before they figure out how your system works. However, if your applications and data are encrypted with proven cryptographic algorithms and the decryption key is not accessible to the attacker, it's game over. At the very least, you have forced the adversary to use a more intrusive method of attack to achieve their objective.

Properly implemented, encryption at rest is designed to protect the confidentiality of your sensitive data from physical access. Encryption can also protect the integrity of the software components on a device, at least so long as you’re either combing it with a cryptographic hash or using an authenticated encryption mode. For example, encrypted storage volumes can prevent attackers from injecting malware, modifying configurations, or disabling security features on a device.

Suffice it to say, using certified and/or industry-standard cryptographic algorithms like AES, RSA, ECC, or SHA will help protect your data at rest and prevent an attacker from gaining access – so long as you keep the secret crypto keys out of reach when the system is powered off (hint: tamper-resistant hardware), during boot, and throughout runtime operation.

Attack Surface Reduction

You might think that more code equals more complexity, making it harder for the attacker to reason over and understand how a system works. But quite the opposite is true. The more code you deploy, the more opportunity an attacker has to find an entry point into the system.

Attack surface reduction is very broad and includes system-level mitigations, the software development lifecycle (SDLC), and data input verification / sanitization. Attack surface reduction is one of the areas where we likely need to layer multiple security approaches, as it’s hard to address everything. Similarly, all these mechanisms are iterative, so the same techniques we’re using and applying to our end device, also need to be applied to our CI/CD pipelines and build systems.

Recall that an attacker only has to exploit one vulnerability to be successful, while the defender must protect against all vulnerabilities. As such, every additional line of deployed code potentially introduces software bugs that an attacker can exploit for their nefarious reasons.

It's a losing battle.

The best approach then is to reduce the attack surface by removing code and interfaces that are not absolutely required, and separating out applications / functions using either hardware or software partitioning. For example, instead of mindlessly deploying a monolithic Linux distribution onto an embedded device, cut out the drivers, features and code you don’t need. A zero-day attack on a graphics card driver can’t be successful on a system that doesn’t include that driver to begin with. Similarly, even a known-vulnerable service cannot be exploited if the service has been disabled or the interface is removed.

Of course, if you need to deploy that graphics driver for functionality, then go for it. Just be careful not to allow unauthorized components to access it, if not necessary, via a principle known as Least Privilege & Mandatory Access Control. Removing drivers and functionality in this context is iterative and can also be leveraged to utilize hardware partitioning.

Hardware Resource Partitioning

If your software stack is allowed unconstrained access to every hardware component on your system, then an attacker can potentially leverage that same access to catastrophic effect. It is like building a boat without bulkheads – a single leak can compromise the whole ship.

Constraining software workloads to particular hardware components (CPU cores, cache, memory, devices, privilege rings, etc.) leads to a cleaner, more straightforward system configuration. It also happens to enable several important security tenets itself such as attack surface reduction and least privilege.

Traditional embedded operating systems have limited protections between processes and application/system dependencies, and since the operating system kernel is similarly not separate from the individual device driver services, the attack surface is large and enables a single exploit to compromise the integrity of the entire system.

An architecture in which components are isolated via strong, hardware-enforced boundaries enables defense-in-depth, especially if interfaces between separated components are tightly controlled. Any vulnerabilities exploited in one application remain constrained to that application, and thus cannot spill over into other components to disrupt the entire system. Furthermore, strict partitioning and isolation can prevent co-execution vulnerabilities, which is an enabling factor for exploit families like Spectre and Meltdown.

Separating components via hardware partitioning, therefore, improves the overall resiliency of the system as one component can no longer directly or indirectly affect another component. Additionally, partitioning the system into discrete components reduces the collective attack surface, and increases overall system security by reducing and/or minimizing privilege escalation, preventing resource starvation, & denial of service, mitigating side-channel and/or timing attacks, and laying the groundwork for future fault-tolerant application approaches.

A similar approach can be applied within the software stack, through containerization and isolation.

Software Containerization & Isolation

Just like one rotten apple can spoil the whole bunch, one insecure piece of code can, if not properly isolated, compromise the entire system.

This is possible because a vulnerability exploited in one piece of code enables the attacker to run arbitrary commands with the same set of privileges as that application – possibly writing to memory or devices where other software components reside. Thus, an initial exploit can quickly gain the attacker unrestricted access to the entire system, or even worse, long-term persistence. Containerization of code helps to mitigate such attacks, preventing an exploit in one component from affecting another.

To mitigate the effects of software exploitation attacks, the defender should containerize, sandbox, and isolate different system functions into separate enclaves. This approach starts at the system architecture stage – ensuring that applications and subcomponents are well-defined and self-contained with clearly understood and enforced boundaries. Next, data flows should be analyzed to ensure that inter-component interactions are known and can be controlled.

Containerization can be accomplished at multiple levels within the software stack, including separate namespaces (i.e. Docker), virtual machines, separation kernels, and/or hardware-enforced memory spaces. When implemented correctly, even exploited software remains constrained to just its process address space, VM, or container thereby limiting the reach of an attacker and preventing the unintended escalation of access across system components.

Of course, the more a defender can do to prevent an exploit from occurring in the first place, the better. One of the best ways to do that is by reducing the system’s attack surface.

Data Input Validation

Many developers fail to imagine how a malicious attacker may intentionally craft malformed inputs that are designed to cause the software to malfunction. As such, data entering a system via any interface can become a vector for attack – exploiting software vulnerabilities to gain unauthorized access or corrupting system/application memory to create a denial of service.

A secure software architecture does not make assumptions about the acceptability of a given input and will validate the format and content of that input before allowing it to be processed by the rest of the system. In other words, inputs from a variety of external sources such as sensors, radios, networks, etc. should be subject to data input validation before use.

Additional vetting of user input (where user means an actual human user, a peripheral user, or a machine operator) is required. But all devices should inspect the conformance of messages to a prescribed data standard as they are passed from device to device.

Furthermore, because any component of the system could become compromised at any point, and thus any message may be maliciously crafted and sent by an adversary, a secure software architecture operates on the principle of mutual distrust. Components within the system must prove their trustworthiness through a continuous (or at least frequent) authentication step. Furthermore, authentication must expire periodically and be reaffirmed.

Device-to-device authentication is often enforced during network formation and at random times thereafter. Message signing and verification are typically included in all messages between authenticated devices.

Validating data before use helps to ensure that external inputs cannot unintentionally interrupt or maliciously exploit system functionality leading to compromise of the system. The majority of malicious data input manipulation attacks target known vulnerabilities in application software and common libraries, which leads us to Secure Software Development practices

Secure SW Dev, Build Options, System Configuration

You’ve probably heard of a buffer overflow attack. It’s a common attack aimed at overwriting memory regions with malicious code. Many compilers can now identify whether such an attack is possible by analyzing your code long before it’s deployed.

Adding this security feature is as simple as configuring your build options correctly.

Of course, other build options can be set to warn you (or error out) on many types of potential security issues and provide security enhancements such as:

Detection of signed/unsigned conversions
Warning for uses of format functions that represent possible security problems
Take advantage of 64-bit address space layout randomization
Compiling code with unintended return addresses
Mitigating variants of Spectre
Defeating stack smashing attacks
Protecting the stack and heap against code execution
Enabling code instrumentation of control-flow transfers
And many more…

Even better, if you have the ability to specify the programming language for your system, you can eliminate entire classes of software vulnerability. For example, the popular Rust programming language can eliminate memory-safety and type-safety programming concerns.

By following defensive coding practices, using secure build options, and configuring the end system for maximum security (depending upon your security requirements), you can significantly decrease the number of possible attacks that can compromise one or more parts of your system.

Secure software build options and system configuration to validated standards are low effort, bare minimum requirements that go a long way toward preventing attackers from driving circles around your other cyber defenses.

Secure and Immutable Configuration

Using our threat model, we should have an idea of what interfaces we need available, and what our deployed system will look like. Using this model, we’ll be able to develop a (recursive) secure configuration for our system. This secure configuration needs to apply to all aspects of our system starting with the hardware, maybe for secure boot and hardware-based keystore. Additionally, we need to ensure the hardware is configured to limit the interfaces / peripherals / memory ranges used by our application, so they aren’t even available to an attacker.

One area that is often neglected, especially relative to hardware, is various test and debug ports used during manufacturing. Guess what? These interfaces can be used by an attacker with physical access in order to bypass your secure configuration. Moving up the software stack, we need to ensure we apply a secure configuration to our operating system. Configuring the OS in a secure fashion is no small feat, and there are lots of nooks and crannies we need to investigate. We need to ensure the kernel is configured securely, access to hardware is limited, ensure system services are started in jails where possible and with restricted permissions, and make sure that containers in use are configured securely . It should be no surprise that most services are not shipped in a secure configuration, and it requires substantial work from the integrator / system developer in order to ensure all doors are figuratively “locked”.

Once the OS is configured securely, we then need to look at our applications. Did we build our applications securely? Do the applications drop unnecessary privileges? As we can see here, a secure configuration requires iteration over multiple components and requires careful identification of the threat model.

Another aspect of a secure configuration is ensuring that configuration remains immutable on a deployed system. So just how would you go about changing a secure configuration? Well, the options here are basically only limited by your imagination, but include: using hardware debug / manufacturing interfaces to change the boot image, replacing the contents of the root filesystem (possibly by replacing the flash device), modifying the kernel command line, adding new driver signing keys to the UEFI SPI flash, mounting a tmpfs on top of a services’ configuration file in order to start the service or application in a less secure fashion, or privilege escalation. Our threat model should generally guide us towards scenarios we need to cover with our design and protections. As we continue to see [e.g. Polkit Vulnerability], privilege escalation is a very real threat and simply put, our systems are not generally designed to prevent an attacker with root-level access from modifying the secure configurations. That isn’t to say there aren’t solutions out there, such as Star Lab’s Kevlar Embedded Security or Titanium Technology Protection which assume an attacker has root-level access when enforcing their security restrictions.

Least Privilege

Can you imagine the chaos that would ensue if everyone in the apartment building had a copy of the master key?

Yeah, your valuables would be gone in a hot second.

The principle of least privilege says that your systems’ software components should only be granted the minimal privileges necessary to do their job, and nothing more. In the context of the apartment building, that means no one else should have the ability to unlock and enter your apartment. Or access the maintenance closet. Or enter the security operations center. In the context of a system, that means applications (and users/operators) should only have access to the minimum set of interfaces and services necessary for their job.

Too often software developers and system engineers take the shortcut – inadvertently (or even explicitly) granting excessive privileges to applications, with an assumption of trusted operator and/or application behavior. That assumption will be quickly invalidated by the attacker.

Instead, embedded systems should be built using Mandatory Access Controls (MAC). Unlike Discretionary Access Controls (which can be modified at-will by users and administrators), systems built upon Mandatory Access Control quantify access grants and restriction policies during system design – controls that are always enforced in the fielded device. As such, there is no user or administrative way to bypass/disable the security controls within the fielded device.

Even if an attacker is successful in compromising a subcomponent of the system or gains root-level access, they will not have a way to modify or disable security settings of the device. When combined with least privilege, Mandatory Access Controls greatly constrain the attacker’s freedom of maneuver, and blocks their ability to modify, disable, or disrupt system services.

To be clear, properly implemented MAC policies do not interfere with normal system operation, and they still allow the system to work as designed and intended. The policies can also be updated in a secure and controlled manner by the system implementer. However, Mandatory Access Control intentionally prevents systems from operating in unintended ways, which is a highly desirable property in embedded computing.

Continuing the theme of limiting assumptions of trust during system development, we’ll now move on to implicit distrust and secure communications.

Implicit Distrust & Secure Communications

If your phone rang right now and somebody you didn’t know asked you to share your credit card number with them, you wouldn’t, right?

Mainly because you don’t trust them.

In much the same way, communication received on your system from external sources should be expressly denied until the remote source has been authenticated. In other words, a secure system doesn’t just let any other system talk to it; it forces external systems to prove themselves. The starting point for secure communication should be default-deny. While default-deny is a good starting principle, we need to make sure we have secure communications in both directions.

More so, just as it is better to share your credit card information to those you trust in a closed room where no one else is around to hear it, your system should enforce secure communication even after the other party has been authenticated.

That typically means data-in-transit will be encrypted, and not just encrypted but also authenticated so it can’t be modified in transit.

Luckily both of these tenets can be implemented using widely used, easily accessible, and proven encryption communications protocols like SSL and TLS with Identity and Certificate management. Of course, anytime crypto is involved, it raises the question of how you plan to protect those TLS keys and certificates (hint: tamper-resistant hardware).

By implementing mutual authentication and encryption, you’ll have more certainty that you are only communicating with trusted entities (and not the attacker) and that nobody else can eavesdrop on what is being communicated.

These mutual authentication and encryption services can and probably should be used for all services including (remote) audit logging, DNS, provisioning, and device monitoring.

Once you are able to securely transmit information from one system to another, you can focus on validating the information sent to prevent malicious data input attacks.

Integrity Monitoring and Auditing

You can’t take action against an attacker if you don’t know when your system is being attacked.

Hence the need for integrity monitoring and auditing.

Integrity monitoring and auditing are important techniques for knowing when a device is being attacked and/or whether it has been compromised. These warnings give you the potential to stop an attacker before it is too late, or at least learn how they exploited your system and what they were able to accomplish after the fact.

Typical techniques include network and OS-level anomaly detection, system log monitoring, and scanning for known malware. These allow the system operator to recognize when some portion of the system may be compromised and take action against the attacker, revoke trust accordingly, or both.

Furthermore, auditing is a requirement of many compliance regulations as the techniques help organizations detect unauthorized modifications to important files, data, or other aspects of your system. HIPAA, NIST, FISMA, NERC, and PCI all require or recommend integrity monitoring and auditing for critical applications and data on distributed systems.

Let’s not forget though, if we’re going to enable remote monitoring of our systems, maybe as part of sending device audit data to a SIEM, we need to configure the offload in a secure fashion. This could be something as simple as mutual TLS authentication of the log client (our device), and the log collector, or something more complex requiring significant engineering effort to implement.

When properly implemented, auditing and monitoring allow you to know when you’ve been attacked, help quantify the damage, and enable you to recover more quickly – preventing lost time, revenue, and damage to your reputation.

Secure Updates

Handling updates in a secure fashion is (not surprisingly) difficult and is also commonly handled incorrectly by many systems. We could probably bore you to death with woes of secure updates going poorly….. Failing to check signatures on update images, not handling user supplied data properly on update scripts providing arbitrary code execution as root, and leaving manufacturing utilities around for exploitation. How we handle updates, let alone in a secure fashion, is going to be dictated by our threat model and the protections we deploy. We may update entire images at once (in which case we can implement signature checking), we may be able to use block-based updates, or we might be able to update independent packages / files. Regardless of how we do it, it’s probably safe to assume that at some point we’ll have to deploy secure updates to our system, whether it’s for updates or increased functionality.

Secure updates need to consider the entire stack, from hardware all the way up to the OS. In almost every case, secure updates are going to require some form of cryptography, which requires us to keep keys out of band from an attacker, and iterate over the tenets again.

No One Tenet to Rule Them All

Unfortunately, there’s no one security tenet to rule them all. There’s no one tip or trick or technology or technique that can immediately and permanently prevent an attacker from compromising your system.

It takes a combination of many, many techniques to do that. Good security practice requires reasoning through potential attacks at every level of the system, understanding and questioning design assumptions, and implementing a layered, defense-in-depth security posture.

Start with these 7 tenets in order to build security into the design, implementation, and operation of your embedded system through the use of:

Threat Modeling – defining our system and possible attacks against it,
Data-at-Rest Protection – authentication and/or encryption to protect applications and data,
Attack Surface Reduction to reduce the amount of code and interfaces that an attacker will have the opportunity to exploit,
Secure & Immutable Configuration – ensuring each component of our system is deployed securely and that an attacker can’t change that configuration
Zero Trust & Least Privilege - to ensure software components can only do what they were intended to do, and nothing more,
Audit & Event Logging - to detect and take action that protects the system against relevant security events.
Secure Update - deploying updates to the system securely, considering impacts to the entire stack

If all these tenets are used and implemented properly on your system, you’ll have a fighting chance against any attacker who seek to exploit your system, steal your IP, or impact your brand reputation.

If you are interested in learning more about these 7 Tenets, check out our free whitepaper! If you want your entire team to learn about these Tenets to improve your security approach, we offer company-specific 60-minute training sessions for free! Click the button below to submit a request to our team.

To learn about what technologies Star Lab can bring to quickly and easily meet your security requirements and protect your system against the full spectrum of reverse engineering and cyber-attacks, contact us.

Schedule a training