Troubleshooting

Enclaver Architecture

Enclaver is a tool that makes it easy to package and run applications in Nitro Enclaves. This document describes the architecture of the CLI, the container-based image format, and the components that run outside and inside the enclave to allow your code to make the best use of the enclave’s security properties.

Different enclave technologies vary in capabilities and deployment patterns. Enclaver currently only supports AWS Nitro Enclaves and this document reflects this deployment pattern. Support for other cloud provider offerings and Intel/AMD enclave features will come in the future.

Enclaver Architecture Diagram

Enclaver CLI

Enclaver’s CLI is designed for two main use-cases:

  1. Build enclave images, sign them and calculate attestations locally on a developer’s machine
  2. Bootstrap and run the Nitro enclave on an EC2 machine

These use-cases directly map to enclaver commands. Refer to the full list of commands to learn about all of the features.

Building an Enclave

enclaver build takes an existing container image of your application code and builds it into a new container image. --file specifies a manifest file for network ingress/egress and runtime parameters of the enclave itself. This manifest is packaged into the image so it can be distributed with the image and included in its cryptographic attestation.

In this example, our manifest file contains the source app container edgebit-containers/containers/no-fly-list (see sample Python app) and the resulting enclave image is saved as no-fly-list:enclave-latest:

$ enclaver build --file enclaver.yaml
 INFO  enclaver::images > latest: Pulling from edgebit-containers/containers/no-fly-list
 INFO  enclaver::images > latest: Pulling from edgebit-containers/containers/odyn
 INFO  enclaver::images > latest: Pulling from edgebit-containers/containers/nitro-cli
 INFO  enclaver::build  > starting nitro-cli build-eif in container: 40bcc4af5c0581c5fb6fc04e2aef4458b326738c7938e08df19244ec3c847972
 INFO  nitro-cli::build-eif > Start building the Enclave Image...
 INFO  nitro-cli::build-eif > Using the locally available Docker image...
 INFO  nitro-cli::build-eif > Enclave Image successfully created.
 INFO  enclaver::build      > packaging EIF into release image
Built Release Image: sha256:da0dea2c7024ba6f8f2cb993981b3c4456ab8b2d397de483d8df1b300aba7b55 (no-fly-list:enclave-latest)
EIF Info: EIFInfo {
    measurements: EIFMeasurements {
        pcr0: "b3c972c441189bd081765cb044dfcf69da0f57050474fb29e8f4f3d4b497cd66567f3f39935dee75d83ea0c9e9483d5a",
        pcr1: "bcdf05fefccaa8e55bf2c8d6dee9e79bbff31e34bf28a99aa19e6b29c37ee80b214a414b7607236edf26fcb78654e63f",
        pcr2: "40bf9153c43454574fa8ff2d65407b43b26995112db4e1457ba7f152b3620d2a947b0e595d513cb07f965b38bf33e5df",
    },
}

The new image contains both the outside and inside components of a secure enclave. This artifact is designed to be handled like a regular container. It can be stored in any container registry and be signed with cosign. Read more about the image format below.

Enclaver will automatically cross-compile to the target architecture of your provided container, which is useful for building on an ARM laptop but running an x86 enclave.

Refer to the full list of commands to learn about all of the features.

Signing an Enclave image

TODO: expand signing instructions. See issue #32.

Running an Enclave

Images built by Enclaver can be run using Docker, or another container runtime. Enclaver-built images come pre-configured with an Entrypoint which will invoke Enclaver’s internal enclaver-run enclave wrapper in order to start the enclave, configure proxying and facilitate logging and monitoring.

enclaver run is provided as a convenience utility for starting Enclaver images in a local Docker Daemon.

All of this happens transparently to you, so the experience you get is very close to running the app outside of an enclave:

$ enclaver run no-fly-list:enclave-latest -p 8001:8001
 INFO  enclaver::run   > starting egress proxy on vsock port 17002
 INFO  enclaver::vsock > Listening on vsock port 17002
 INFO  enclaver::run   > starting enclave
 INFO  enclaver::run   > started enclave i-00e43bfc030dd8469-enc1840fa584262e1a
 INFO  enclaver::run   > waiting for enclave to boot
 INFO  enclaver::run   > connected to enclave, starting log stream
 INFO  enclave         >  INFO  enclaver::vsock > Listening on vsock port 17001
 INFO  enclave         >  INFO  enclaver::vsock > Listening on vsock port 17000
 INFO  enclave         >  INFO  odyn::enclave   > Bringing up loopback interface
 INFO  enclave         >  INFO  odyn::enclave   > Seeding /dev/random with entropy from nsm device
 INFO  enclave         >  INFO  odyn            > Enclave initialized
 INFO  enclave         >  INFO  odyn            > Startng egress
 INFO  enclave         >  INFO  odyn            > Startng ingress
 INFO  enclave         >  INFO  enclaver::vsock > Listening on vsock port 8001
 INFO  enclave         >  INFO  odyn            > Starting KMS proxy
 INFO  enclave         >  INFO  odyn::kms_proxy > Generating public/private keypair
 INFO  enclave         >  INFO  enclaver::vsock > Connection accepted
 INFO  enclave         >  INFO  enclaver::vsock > Connection accepted
 INFO  enclave         >  INFO  odyn::kms_proxy > Fetching credentials from IMDSv2
 INFO  enclave         >  INFO  odyn::kms_proxy > Credentials fetched
 INFO  enclave         >  INFO  odyn            > Starting ["python", "-m", "flask", "run", "--host=0.0.0.0", "--port=8001"]
 INFO  enclave         >  * Serving Flask app "/opt/app/server.py"
 ...your app logs...

Output from the application is automatically logged by the “wrapper” container. When implementing an enclave application you should carefully consider what is logged, and avoid logging anything which is not intended to leave the confines of the enclave.

enclaver run --debug starts the underlying Nitro Enclave in debug mode, and automatically gathers the output of the underlying VM’s console into the wrapper container logs. This is intended for debugging issues related to attestations and communicating with services outside the enclave, and not for general debugging. For debugging during development, it is more useful to run your container directly outside of an enclave.

Refer to the full list of commands to learn about all of the features.

Enclaver Image Format

The Enclaver image format is a regular OCI container image consisting of:

  1. A base image built on amazonlinux, with nitro-cli installed (this may be slimmed down in the future)
  2. An Enclaver “wrapper” binary, installed at /usr/local/bin/enclaver
  3. Enclave-specific application.eif and enclaver.yaml files installed under /enclave/

The EIF file is in an AWS-specified format, and contains a kernel and Linux userland including the enclave application and Enclaver’s “inner” component, odyn.

The enclaver.yaml manifest file is an exact copy of the one that was used to build the image. A second copy of this file is bundled into the filesystem within application.eif, both for use by odyn for policy enforcement, and so that it is covered by cryptographic attestations.

Enclaver-built images are configured with an ENTRYPOINT which will automatically launch Enclaver’s outer proxy and enclave supervisor, which in turn launches the enclave and inner proxy and supervisor, which in turn launches your application.

To minimize the attack surface of enclave applications, it is not possible to pass runtime parameters.

Calculating Cryptographic Attestations

enclaver build outputs the cryptographic attestation of an image. An attestation is a reproducable “measurement” of a piece of code that can be used to give the code a unique identity. The word “measurement” is used because, just like a ruler, Enclaver records the content of various parts of the code that make up the enclave image. The hash of this measurement is recorded into Platform Configuration Registers (PCRs). A collection of certain PCRs (eg. PCR0-4 + PCR8) is the unique attestation of that particular piece of code.

TODO: Implement enclaver trust command. See [issue #38](https://github.com/edgebitio/enclaver/issues/38).
$ enclaver trust registry.example.com/my-app:v1.0
{
  "Measurements": {
    "HashAlgorithm": "Sha384 { ... }",
    "PCR0": "7fb5c55bc2ecbb68ed99a13d7122abfc0666b926a79d5379bc58b9445c84217f59cfdd36c08b2c79552928702efe23e4",
    "PCR1": "235c9e6050abf6b993c915505f3220e2d82b51aff830ad14cbecc2eec1bf0b4ae749d311c663f464cde9f718acca5286",
    "PCR2": "0f0ac32c300289e872e6ac4d19b0b5ac4a9b020c98295643ff3978610750ce6a86f7edff24e3c0a4a445f2ff8a9ea79d",
    "PCR8": "70da58334a884328944cd806127c7784677ab60a154249fd21546a217299ccfa1ebfe4fa96a163bf41d3bcfaebe68f6f"
  }
}
EIF Info: EIFInfo {
    measurements: EIFMeasurements {
        pcr0: "b3c972c441189bd081765cb044dfcf69da0f57050474fb29e8f4f3d4b497cd66567f3f39935dee75d83ea0c9e9483d5a",
        pcr1: "bcdf05fefccaa8e55bf2c8d6dee9e79bbff31e34bf28a99aa19e6b29c37ee80b214a414b7607236edf26fcb78654e63f",
        pcr2: "40bf9153c43454574fa8ff2d65407b43b26995112db4e1457ba7f152b3620d2a947b0e595d513cb07f965b38bf33e5df",
    },
}

An attestion must be reproduceable in order to ensure that each time the enclave is started, and nothing about it has been modified, the attestation remains constant. Other systems will rely on this reproduceability to build trust with the software.

If an attestation matches, engineers can guarantee that its configuration is exactly what was tested/verified/trusted. If you’re remotely communicating with an enclave, the attestation can remotely prove it’s configuration. An attestation serves as a piece of identity that can’t be hijacked because it’s crypographically tied to the code itself.

Register Description
PCR0 Measures the length of our EIF file. Since it’s critical that our enclave code is not modified, this is important to check.
PCR1 Measures our enclave’s kernel and boot RamFS data. Again, we don’t want our kernel image modified or the kernel parameters changed.
PCR3 Measures the IAM instance role assigned to your EC2 machine.
PCR4 Measures the instance ID of a specific EC2 machine.
PCR8 Measures the enclave image file signing certificate.

When using attestations, you must decide which “measurements”, the PCR values, are useful to you. In almost all cases you should care about the image and kernel parameters, but if you’re running multiple enclave instances with the same code, PCR4 will not be useful because it will be unique to each EC2 machine running your enclaves.

Enclaver’s trust command uses PCR0, PCR1, PCR2, PCR8 to calculate its attestation before execution. Because it’s not possible to know PCR3 until the machine is running, it is most useful when configuring an AWS Key Management Service (KMS) policy.

From within the enclave, the get-attestation-document API also provides the ability for your code to query its own attestation document.

Components Outside the Enclave

The goal of components outside of the enclave are to monitor the health of the enclave and to route allowed traffic into the enclave. Since isolation is a critical component to enclave security, Enclaver has proxies sitting on both sides of the virtual socket (vsock) that connects the inside and outside.

These components have minimal overhead compared to the CPU and RAM carved out for the enclave itself.

Enclave Supervisor

enclaver run is the enclave supervisor. It runs as a systemd unit and exits if the enclave dies. By design, there is very little visibility into the enclave, so the command watches the context ID (CID) for information provided directly from the Nitro hypervisor.

[Unit]
Description=Enclaver
Documentation=https://edgebit.io/enclaver/docs/
After=docker.service
Requires=docker.service
Requires=nitro-enclaves-allocator.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker exec %n stop
ExecStartPre=-/usr/bin/docker rm %n
ExecStartPre=/usr/bin/docker pull registry.edgebit.io/no-fly-list:enclave-latest
ExecStart=/usr/bin/docker run \
    --rm \
    --name %n \
    --privileged \
    --device=/dev/nitro_enclaves:/dev/nitro_enclaves:rw \
    -p 8001:8001 \
    registry.edgebit.io/no-fly-list:enclave-latest

[Install]
WantedBy=multi-user.target

Outer Proxy

The outer proxy sets up routing from the rest of your AWS infrastructure into the enclave. The other end of the virtual socket is running within the trusted environment, which protects against a malicious outer proxy and enforces the enclave’s network policy.

The outer proxy only forwards HTTP and TCP traffic into the enclave.

If the enclave is running in debug mode, the outside proxy allows for streaming logs through the virtual socket for debugging.

Components Inside the Enclave

The goal inside of the enclave is to protect your workload from the outside world. A single component, named odyn, provides all of the inner functionality.

Process Supervisor

Enclaver runs the supervisor as PID2 (soon to be PID1) inside the enclave to accomplish:

  1. Enclave bootstrap - bring up loopback and seed entropy
  2. Execute the original ENTRYPOINT from your container
  3. Provides the entrypoint status to the outside
  4. Forwards the logs to the outside
  5. Reaps zombies (disabled until running as PID1)

Inner Proxy

The inner proxy provides routing to the outside world and does network filtering based on the policy baked into the enclave image. This protects your code from outside network based attacks and is a layer of defense against exfiltration of data caused by a vulnerability in a library inside the enclave.

For ingress, TLS is terminated and ingress policy is enforced. The private keys used for TLS termination are fetched from KMS.

For egress, policy is enforced before traffic leaves the enclave.

The host hostname can refer to localhost on the parent instance of the enclave, which is useful for egress traffic to stay local to the machine, like talking to other containers running outside the enclave.

The inner proxy can optionally append the attestation of the enclave to Decrypt, GenerateDataKey, and GenerateRandom calls to AWS KMS, which allows for super easy integration for your code to use your KMS keys to decrypt data within the enclave. This is when you see the power of using the output from enclaver trust --kms as part of a KMS key policy.

TODO: update with final enclaver trust command. See issue #38.

Verifying Cryptographic Attestations

TODO: Implement this feature. See issue #35.

enclaver run --verify-before-run attestation.json will verify an attestation of an image after fetching it, but before executing it. If the comparison fails, the violating PCRs will be logged and the command will fail with an exit code. Since our threat model can consider the host hostile, this is more of a corruption check.

Inside of the enclave, the KMS proxy will also fetch the attestation, but it will come directly from the hypervisor, so it can be fully trusted. The get-attestation-document API is only available inside of the enclave.

TODO: expand general usage with other non-KMS systems