SGX - is HeapMaxSize necessary?

[Fredrik_T_]

The .edl files contain a HeapMaxSize entry. The SDK User Guide states that this is because

Enclave memory is a limited resource. Maximum heap size is set at enclave creation.

But doesn't the SGX specification allow EPC page swapping (EPA, EBLOCK, ETRACK, EWB)?

Or in a more practical sense: is there a disadvantage to setting HeapMaxSize=2^64 Bytes?

Maybe EPC page swapping is not yet supported by the SDK, or maybe the trusted enclave code has to manually trigger such swapping?

[Simon_J_Intel · Accepted solution]

The first generation of enclave instructions does not support the addition of pages to the enclave after EINIT instruction has been executed. All memory used by the enclave must be EADDed to the enclave prior to EINIT.

The EWB/ELD instructions deal with adding and removing pages from the limited physically protected memory (EPC) to regular memory on the platform, they do not expand the size of the enclave, but on Windows they are not currently used. As EPC memory is limited to ~90MB on 6th Generation Core Processor the ramifications of setting HeapMazSize=2^64 would result in an unloadable enclave.

[Fredrik_T_]

Thank you Simon, good to know that EWB/ELD are not currently used.

The SGXv2 specification seems to contain all the parts necessary to avoid static and limited enclave sizes. I hope future SDK iterations can bring virtual memory benefits for enclaves:

1. unlimited enclave sizes (EPC page swapping, which requires OS support for EWB etc.) and
2. dynamic enclave sizes (SGXv2).

I have a couple of follow-up questions:

1. I read between the lines that the SDK currently does not expose SGXv2 functionality (no dynamic page allocation via EAUG) - is that correct?

2. Also, do you know the reason why EPC memory is limited to ~90MB?

According to the SGX programming reference, the BIOS sets aside a range of Processor Reserved Memory in which the EPC resides. A memory encryption engine is given as one implementation example, which encrypts all EPC pages that leave the processor boundary.

Are the 90MB a BIOS limitation, or do the current processors not use MEE+regular DRAM, but instead use dedicated EPC memory within the CPU package?

It seems that in some Dell Laptop's BIOS the EPC size can be set as high as 128MB? (https://jbeekman.nl/blog/2015/10/sgx-hardware-first-look/)

[Simon_J_Intel]

The 6th generation core processors that support SGX only support the v1.0 instruction set. The addition functionality in the v2 set are not yet available.(I have no statement on the intercept of the v2.0 instructions at this time).

Enclave Page Cache (EPC) is a subset of Processor Reserved Memory (PRM), BIOS reserves PRM and then the HW carves out EPC from that region, depending on the amount of metadata the MEE requires. Our tutorial slides (slide 163 onwards) contains a description of the MEE. So when the BIOS reserves 128MB PRM the EPC carve out is ~90MB.

[Fredrik_T_]

Thank you for your prompt answers Simon. I'll attempt to summarize:

Current hardware only supports the v1 instruction set.

v1 specifies instructions that support page swapping between (encrypted) EPC and (regular) main memory. Neither Windows 10 nor the SGX SDK currently support this paging feature.

Current hardware can only set aside a limited amount of PRM (~128MB), and consequently a limited amount of EPC (~90MB).

Due to the current lack of paging support (and lack of dynamic page allocation that v2 will provide) this means that the combined HeapMaxSize of all enclaves loaded at the same time cannot exceed said ~90MB.

Future hardware may set aside a larger EPC; this is not limited in any way by the SGX spec.

[Simon_J_Intel]

That summary is accurate.

[Jonathan_D_1]

Hi Simon,

I'm trying to determine whether I understand the SGX memory limitations as of SDK version 1.9.100.39124. In particular, I've seen many somewhat dated sources (this page included) referring to a 90MB limit that prevents enclaves from using much larger heap sizes. However, on my Ubuntu 16.04 system using the SDK version I listed above, I'm able to get heap sizes of, say, 8 GB on a machine with 16 GB of RAM. In one test I was able to use "malloc()" within an ecall to successfully allocate a gigabyte of memory, write to it, and read from it. That leads me to wonder whether the SDK now supports larger heaps than it used to, but I haven't been able to confirm this in any documentation provided with the SDK or online. Can you confirm this, or explain why my testing yields the above behavior?

[JohnMechalas]

Hi Jonathan,

The current Linux implementation for Intel SGX supports EPC paging, which means your heap size is not limited to the max EPC size.