Troubleshooting

Running the No-Fly-List app inside an Enclave

This guide will walk through running the No-Fly-List app, which checks passengers attempting to fly on an airline against a no-fly list. It’s a fairly simple Python application that requires protection “in-use” for it’s data, because we don’t want anyone to be able to see the full no-fly list. The app uses a secure enclave and Amazon KMS to achieve that.

No-Fly demo on YouTube

How does the enclave's isolation protect the no-fly list?

The enclave is extremely isolated by design. It guarantees that no one can inspect it’s memory, interactively log in to it, or read data inside of it. This makes it a safe place to decrypt our data and run the passenger matching logic against it. When auditing the code, we can see that it returns our allowed or denied message and nothing more. There are no avenues where the no-fly list can escape.

How does the enclave's code identity protect the no-fly list?

The enclave has a code identity, called an attestation, that unqiuely identifies it. This is generated through cryptography, so it’s impossible to fake. Since it’s code, it’s not possible to steal, unlike a human’s identity.

enclaver trust will show you the cryptographic attestation of a specific enclave image. Our specific code, via the attestation, is granted access to read our encryption key needed to decrypt the no-fly list. With a locked-down access policy, it’s impossible for anything other than this specific code to read the key.

Here’s an example of an attestation:

TODO: Implement trust command. See issue #38.

$ enclaver trust registry.edgebit.io/no-fly-list:enclave-latest
TODO: add real attestation

In the recent past, there was an incident – this is a rumor – that caused the entire cast of Sesame Street to be added to the no-fly list. We can find out if that’s true :)

For this example you’ll need an EC2 instance with support for Nitro Enclaves enabled (c6a.xlarge is the cheapest x86 instance) and Docker installed. See the Deploying on AWS for more details.

CloudFormation

The No-Fly-List App

On startup, the app fetches an encrypted blob from S3. This blob is encrypted with a scheme called “envelope encryption”. Let’s take a quick detour to understand it, because it’s both interesting and really useful.

Envelope Encryption

Envelope Encryption involves encrypting our data twice, once at the app-level with key A, and then encrypting key A with key B. In our app, we have the data that was encrypted with key A (the data key), and an encrypted version of key A. This is our “envelope”, because we’ve wrapped our data with the second level of encryption using key B (master key).

{
  "key-A-ciphertext": "UhOUXl...besRT=",
  "data-ciphertext":  "QZQE2J...uypwE="
}
Why do we use Envelope Encryption?

If we throw away the plaintext of key A, the envelope only contains encrypted data, so it can be stored in a database or sent through microservices safely. If a consumer down the line needs to decrypt it, they can decrypt key A if they have access to key B. The process unlocks numerous benefits:

  1. Data keys (key A) can be used to encrypt multiple files or pieces of data received at the same time. It’s common to have a database row that has multiple fields encrypted with a single key (“field-level encryption”):
    ID Encrypted Data Key First Name Last Name Address
    1 UhOUXlT2an029Xqva… kDAgEQgDu… vQFPsDGU… QE2J4n…
  2. Encrypted data keys are stored with the data, so there isn’t additional key management needed to store and track them.
  3. Envelope encryption normalizes using unique keys for your data instead of one single key that can be compromised.
  4. Different encryption schemes can be used to maximize performance. Data keys typically need to encrypt large objects, so they need to be performant and use symmetric key algorithms. Using public key algorithms are slower but carry the convinience of the public/private key separation. Envelope encryption allows using both schemes where each are well-suited for the task.
  5. Possible to encrypt the same data key under multiple master keys without re-encrypting the raw data.

Functionally, this means we’ll have two layers of encryption, the first is encrypted using AES with a key unique for each sensitive item and then the unique keys are encrypted by a master key that is stored in AWS KMS.

Here’s what the actual envelope used by the app looks like:

{
  "data-key-ciphertext-base64": "AQIDAHgFXi2TEB5uhOUXl62UNxtALVzp0EqotGT2an02XqvQvQFPsDGUMVCbesRTBymyEYYBAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMn/xg+FHWxztbsikDAgEQgDulQ5ROICb+58HcwXTls2bUohxdxN4FFZnp4QFbAweKGJwEEmhkNp7HnrQU+wPUXvQVc7m+bPVeoXksSQ==",
  "aes-ciphertext-base64": "U2FsdGVkX18a+Ji6uIdSAc9GQF3BV1EqZQE2J42nOJUxiyDJr12mSXI2qm5Z5no1KZGM4dKeuBSDwQuyOCJrpwE0g6+XERruQLdazh02Vq3VLx5MwaM7pVBwJLXlt6Wnl8HWtXNjNCySQKrMJmUIH+arCxthxUho4ABiNZ+nJEW3+GEYsmD92KcK/CzytFJVH6X8QajJn4kq5dbMa6rDxw=="
}

Decrypting the No-Fly List

Inside of the enclave, we have a policy that allows us to decrypt the data key. Since it’s recorded in the file, it’s easy to know which one to request from KMS. When we get that, we can use it to decrypt the main part of our file.

While we don’t explore it here, the powerful part is that we could locally decrypt large or numerous files without having to transmit the ciphertext through KMS, because we have the plaintext data key already. The isolation guarantees of the enclave make it safe to cache the plaintext data key, unlike if you were keeping it in RAM or on disk in a regular VM.

The concept of multiple data keys is called “field level encryption” and passing around these envelopes around is “app-level encryption”.

Hope you learned something! Let’s jump into deploying our app.

Create the Enclave Configuration

Enclaver builds enclave images based on a configuration file, which specifies the container that holds the app code, the network policy for egress, and a few other details. This policy is packaged into the image because it is distributed with the image and included in its cryptographic attestation.

Here’s the configuration for the No-Fly list app:

version: v1
name: "test"
target: "no-fly-list:enclave-latest"
sources:
  app: "registry.edgebit.io/no-fly-list:latest"
defaults:
  memory_mb: 4096
kms_proxy:
  listen_port: 9999
egress:
  allow:
    - kms.*.amazonaws.com
    - s3.amazonaws.com
    - 169.254.169.254
ingress:
  - listen_port: 8001

It’s pretty straightforward. The sources.app parameter specifies the source container for our code. Since we’re using AWS KMS for cryptography and S3 for fetching our encrypted no-fly list, those addresses are allowed. The IP address is the AWS instance metadata service, which grants dynamic credentials to use for the KMS and S3 requests.

Build the Enclave Image

enclaver build takes an existing container image of your application code and builds it into a new container image with enclave-specific components added in. This is what we’ll run on our EC2 machine. This image can be pushed to a registry like any other container.

We’re passing in our manifest file from above to the build:

$ enclaver build --file enclaver.yaml
 INFO  enclaver::images > latest: Pulling from edgebit-containers/containers/no-fly-list
 INFO  enclaver::images > latest: Pulling from edgebit-containers/containers/odyn
 INFO  enclaver::images > latest: Pulling from edgebit-containers/containers/nitro-cli
 INFO  enclaver::build  > starting nitro-cli build-eif in container: 40bcc4af5c0581c5fb6fc04e2aef4458b326738c7938e08df19244ec3c847972
 INFO  nitro-cli::build-eif > Start building the Enclave Image...
 INFO  nitro-cli::build-eif > Using the locally available Docker image...
 INFO  nitro-cli::build-eif > Enclave Image successfully created.
 INFO  enclaver::build      > packaging EIF into release image
Built Release Image: sha256:da0dea2c7024ba6f8f2cb993981b3c4456ab8b2d397de483d8df1b300aba7b55 (no-fly-list:enclave-latest)
EIF Info: EIFInfo {
    measurements: EIFMeasurements {
        pcr0: "85aaa37e85a0b7178bb5700a8c1ae584bf4f994996db6f18503e215cf35b65f737b19e822b3f10eb634317bd4f11deee",
        pcr1: "bcdf05fefccaa8e55bf2c8d6dee9e79bbff31e34bf28a99aa19e6b29c37ee80b214a414b7607236edf26fcb78654e63f",
        pcr2: "cb64e00fce6987d7484c18cc4c19d92ec80955d86f6b43b2d4794f9edc1a9d0200d72cf3e876566e9d888bc971413f46",
    },
}

Run the Enclave

enclaver run executes on an EC2 machine to fetch, unpack and run your enclave image. First, SSH to your EC2 machine:

$ ssh ec2-user@<ip address>

After the image is fetched, it is broken apart into the outside and inside components. The outer components are started first, then the enclave, with the inner components inside, is started.

Note: you must use the container image shipped by EdgeBit for the demo to work correctly. The build above is shown for educational purposes but your build will cause the PCR measurements to differ from the key access policy.

We will start it manually using Docker, but you can also set up a systemd unit.

$ docker run \
    --rm \
    --detach \
    --privileged \
    --name enclave \
    --device=/dev/nitro_enclaves:/dev/nitro_enclaves:rw \
    -p 8001:8001 \
    registry.edgebit.io/no-fly-list:enclave-latest

Check to see that the enclave was run successfully:

$ docker logs enclave
 INFO  enclaver::run   > starting egress proxy on vsock port 17002
 INFO  enclaver::vsock > Listening on vsock port 17002
 INFO  enclaver::run   > starting enclave
 INFO  enclaver::run   > started enclave i-00e43bfc030dd8469-enc1840fa584262e1a
 INFO  enclaver::run   > waiting for enclave to boot
 INFO  enclaver::run   > connected to enclave, starting log stream
 INFO  enclave         >  INFO  enclaver::vsock > Listening on vsock port 17001
 INFO  enclave         >  INFO  enclaver::vsock > Listening on vsock port 17000
 INFO  enclave         >  INFO  odyn::enclave   > Bringing up loopback interface
 INFO  enclave         >  INFO  odyn::enclave   > Seeding /dev/random with entropy from nsm device
 INFO  enclave         >  INFO  odyn            > Enclave initialized
 INFO  enclave         >  INFO  odyn            > Startng egress
 INFO  enclave         >  INFO  odyn            > Startng ingress
 INFO  enclave         >  INFO  enclaver::vsock > Listening on vsock port 8001
 INFO  enclave         >  INFO  odyn            > Starting KMS proxy
 INFO  enclave         >  INFO  odyn::kms_proxy > Generating public/private keypair
 INFO  enclave         >  INFO  enclaver::vsock > Connection accepted
 INFO  enclave         >  INFO  enclaver::vsock > Connection accepted
 INFO  enclave         >  INFO  odyn::kms_proxy > Fetching credentials from IMDSv2
 INFO  enclave         >  INFO  odyn::kms_proxy > Credentials fetched
 INFO  enclave         >  INFO  odyn            > Starting ["python", "-m", "flask", "run", "--host=0.0.0.0", "--port=8001"]
 INFO  enclave         >  * Serving Flask app "/opt/app/server.py"
 ...app logs...

Submit Passenger Names

Now the fun part. Let’s see who can fly and who can’t. Remember, a key part of this scenario is that no one should be able to see the complete no-fly list, but we should get back an answer for each person when they buy a ticket.

We know that members of Sesame Street might not be allowed to fly. Test it out for yourself from the EC2 machine:

$ curl localhost:8001/enclave/passenger?name=foo
foo is cleared to fly. Enjoy your flight!

See how many names you can discover that won’t be flying today.

Explore the Enclave

KMS Policy

The beginning of the guide discussed how the crypographic attestation is used to protect the key. This configuration happens within AWS KMS on the key access policy. The policy for the No Fly List app uses the PCR0 measurement, which measures all of the enclave code, so it represents the most holistic measurement of our enclave image.

Enclaver’s KMS integration attaches the full attestation to KMS requests automatically. The measurement for PCR0 gets used in the policy like this:

"StringEqualsIgnoreCase": {
    "kms:RecipientAttestation:PCR0": "85aaa37e85a0b...17bd4f11deee"
}
View the complete access policy

The Principal may stand out to you — it allows any AWS account to use this key. Anyone can run this demo, so anyone can use the key…well, anyone inside of this exact Enclave image can use the key.

For real use-cases, you would lock down this policy to your AWS accounts and you could target higher PCR values for greater specificity.

{
    "Version": "2012-10-17",
    "Id": "key-noflylist",
    "Statement": [
        {
            "Sid": "Allow public use of the key for no-fly-list demo",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "StringEqualsIgnoreCase": {
                    "kms:RecipientAttestation:PCR0": "85aaa37e85a0b7178bb5700a8c1ae584bf4f994996db6f18503e215cf35b65f737b19e822b3f10eb634317bd4f11deee"
                }
            }
        }
    ]
}

Dedicated CPUs

If you booted a c6a.xlarge, the full machine has 4 vCPUs. By default, Enclaver dedicates 2 of those to the Nitro Enclave. Dedicated CPUs are part of the isolation and protection of your workloads in the enclave.

You can test this out by running top and then hitting 1 to show a breakdown of CPUs. Notice that CPUs Cpu1 and Cpu3 are missing here:

top - 14:48:27 up 6 min,  1 user,  load average: 0.01, 0.06, 0.03
Tasks: 111 total,   1 running,  52 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7949808 total,  6388256 free,   669472 used,   892080 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  7046532 avail Mem

Check out the Code

This application is on GitHub: https://github.com/edgebitio/no-fly-list/blob/main/server.py

Once you factor out the S3 fetching and the boilerplate KMS handling, our actual logic is just a handful of lines that is easily audited. This is the ideal type of enclave app. It’s focused, simple and acts like a secure sidecar to the rest of our app.

Next Steps

This example walked through running a simple Python app that represented running a specific microservice or a security-centric function within an enclave. It’s also possible to run an entire application in an enclave to wrap it in a higher level of security.

Check out running Hashicorp Vault for a walkthrough.