This post summarizes a paper the EdgeBit team published and presented at the Software Supply Chain Offensive Research and Ecosystem Defenses (SCORED ‘23) event at ACM’s 2023 conference on Computer and Communications Security in Copenhagen.

As a security team, it is not realistic to close out every security issue across your apps and infrastructure. Understanding your risk and tracking your list of known vulnerabilities is attainable however. This goal is shared by the Federal government in cybersecurity regulations kicked off by the 2021 executive order.

Today, EdgeBit’s security platform tracks vulnerabilities in your apps, aided by our unique runtime context to remove noise, to keep your developers on top of issues right in the pull request or in their Jira backlog for a future sprint.

But…can we go further?

Fine-grained tracking of supply chain risk

Tracking risk is complex because both long and short running workloads may change state over time. At build-time, it’s typical to generate SBOMs and sign containers. But at run-time, tools like Open Policy Agent and Kyverno only check the signatures or Kubernetes Pod attributes, not the full contents.

Our hypothesis was simple but potentially technically complex — can we go beyond only checking container signatures and check an entire container against its software bill of materials (SBOM) in real time?

If you can guarantee that your SBOM faithfully represents your workload, it’s a perfect proxy for understanding risk and enforcing policy.

Our investigation focused on features in the Linux kernel that promote system integrity:

1. Integrity Measurement Architecture (IMA)
2. Extended Verification Module (EVM)
3. Linux Security Modules (LSM)
4. Extended Berkeley Packet Filter (eBPF)

The gravitation towards this set of tools is driven by a desire to not require a reboot nor require a non-standard kernel module to be installed. This would allow end users to protect machines via a Kubernetes DaemonSet or systemd unit without the need to tweak an operating system image or use a custom kernel.

Evil Maid style attacks are not in scope for this effort, although protections exist that aren’t covered here.

Our goal: Frictionless adoption at-scale

Adoption of the proposed SBOM enforcement scheme is predicated on low friction for development teams. This scheme piggy-backs of the increasing adoption of SBOMs and the ability to match them to a deployable artifact like a container or Linux package.

SBOMs provide app inventory down to the file level and serve as a compliance artifact. The blessed app contents are easy to encode into the container by mutating the extended attributes and map directly back to the SBOM, which itself maps back to the source code.

The process of blessing container images or files is done by recording the digest or signature in the file’s extended attributes as part of the build pipeline. The result is a container that can be stored in any registry and be run normally.

The in-toto attestation framework allows for these imported SBOMs to show how they were derived, which chains together all of the elements in your supply chain.

In production, having all of the enforcement mechanisms within the Linux kernel allows for an extra degree of trust without sacrificing performance. All it takes is the ability to launch a privleged DaemonSet on Kubernetes, which means no manipulating the underlying operating system, preserving the so-called “golden images”.

Next, we’ll dive into the technology to make this scheme possible. Spoiler: there are limitations laid out below that prevented us from realizing the full vision in the current Linux kernel.

IMA and EVM for filesystem integrity

Linux Integrity Management Architecture (IMA) is a lesser known feature of the Linux kernel which is used to measure and enforce file contents in a high-performance manner. Introduced in Linux 2.6.30, IMA makes use of filesystem extended attributes to record a digest or signature and then checks it against the file contents whenever the file is accessed. This is only useful for static files however, since legitimate modification of file contents (e.g., updating /etc/resolv.conf) would cause the IMA digest/signature to fail.

While it’s possible to write an IMA policy which protects a subset of files while still allowing the rest to be modified, it wouldn’t provide much security since an actor who can maliciously modify file contents can similarly modify extended attributes to match. The extended attributes also need to be protected.

Filesystem extended attributes are typically protected by Extended Verification Module (EVM), which compares a subset of extended attributes against a digest or signature. This mechanism isn’t always included in kernel builds though, and isn’t currently present in Google’s Container-Optimized Operating System or Amazon Linux.

Here’s an example of manipulating EVM manually:

$ evmctl ima_hash ...  # calc and store in xattr security.ima

$ evmctl sign ...      # create sig from private key, store in security.ima

$ evmctl ima_verify    # verify sigs

Here’s a summary of the pro’s and con’s of IMA and EVM together:

• Highly performant via kernel
• IAM is portable to most Linux distros
• EVM provides tamper resistance that IMA lacks by itself

• IMA is only useful for static files
• EVM is not well supported in Google’s Container-optimized OS or Amazon Linux

Where does that leave us? The lack of distro support for EVM is a huge blocker. But, there exists another way to potentially limit modification of extended attributes: eBPF.

Protecting from modification with eBPF and LSM hooks

Extended Berkeley Packet Filter (eBPF) is a facility of the kernel which allows code to be loaded and run within the context of the kernel itself. We’ve covered eBPF before on the blog and it’s pretty incredible — you can make powerful extensions to machine logic at low levels — all without requiring a reboot or messing with the base image. Linux Security Modules (LSM) hooks provide an access control framework in the kernel which is frequently used with eBPF and SELinux.

For SBOM enforcement, we want to verify and protect the contents of files inside our container, as stored in the extended attributes during our build. The goal is to prevent malicious overwrites, chained exploits from executing and manual changes if an engineer is logged into production.

Here’s an example program to prevent removal of security. extended attributes:

#define PREFIX "security."
SEC("lsm/inode_removexattr")
int BPF_PROG(protect_xattrs, void *idmap, void *dentry, const char *name)
{
   char kname[sizeof(PREFIX) + 1];
   if (bpf_probe_read_kernel_str(&kname, sizeof(kname), name) == 0)
       return -EPERM;
   if (has_prefix(kname, PREFIX))
       return -EPERM;
   return 0;
}

In order to ensure the integrity and stability of the system, a number of constraints are put in place that limit what eBPF code can do at runtime. The major limitation encountered in our experimentation is that while protecting extended attributes, eBPF does not have a helper function for reading extended attributes from a given inode.

If such a helper did exist, a simple implementation of this scheme could look at the extended attributes to see if any of them belong to the security class, and deny the file mutation if so.

Pointer-chasing from a given inode might be an option, if it weren’t for the fact that extended attributes are filesystem-specific and require calling into filesystem-specific code — disallowed in eBPF’s context.

Calling out to a userspace process which does the extended attribute lookup also proved to be a dead end, since LSM hooks don’t have a mechanism for synchronous communication. There has been some interest by the development community in adding this functionality, but progress has been slowed by concerns from upstream developers about the security impacts of such a change. Assuming that these issues are addressed though, the full story can be realized.

Summarizing eBPF + LSM, we get really close to our ideal vision:

• Highly performant via kernel
• Easy to install via binary, container or DaemonSet
• Supported in popular distros

• Too restrictive around Linux Security Module (LSM) hooks

Making SBOM enforcement possible

Enforcing the integrity of an application workload through its SBOM allows security teams to use the SBOM as a representative of run-time state. SBOMs enriched with vulnerability information on a continual basis provide an accurate picture without requiring invasive monitoring of the running workloads.

As laid out in the pro/con lists above, this scheme is not possible today without changes to eBPF helper functions.

Our research has shown that all of the software supply chain tools exist to support this behavior and that the series of Linux kernel modifications required to make the proposed scheme complete are technically viable. It remains to be seen if upstream consensus can be reached to extend eBPF’s ability to reach extended file attributes or allow synchronous communication with LSM hooks.

Adoption of this scheme is uniquely ideal for large scale deployment because of the universal APIs offered by kernel-level enforcement and eBPF’s ability to modify Linux behavior without requiring custom patches or a heavily customized golden OS image.

Getting the benefits of SBOMs for risk analysis

The good news is that today, SBOMs enriched with vulnerability data and realtime context can already level up your security posture. SBOMs processed through EdgeBit’s security platform provide risk analysis and noise reduction for both developers and security engineers.

Using SBOMs as a common denominator, EdgeBit protects complex software supply chains regardless of the frameworks and ecosystems your development teams use.

We hope to drive forward the changes required to achieve this extra level of run-time enforcement in future Linux kernel releases. Look for more technical research projects from the EdgeBit team in 2024.