SGX protected memory limit in SGX

[Feng_C_]

Hi

The SGX SDK reference indicated the SGX has limited protected memory as 128 MB. But I just found the memory limit can be extended almost up to 4GB in Linux platform (I set the HeapMaxSize = 0xF0000000 = 3.75GB). In addition, I allocated five 32 MB buffer array in the program (total 160 MB). The program run successfully.

So, I just wanna make sure if all of the 4GB are protected memory?

Thanks.

Feng

[Surenthar_S_Intel]

Hi Feng,

The limit of 128 MB comes from the BIOS. To go beyond that, there needs to be paging support. The Linux driver supports paging, but Windows does not.

Thanks and Reagrds,
Surenthar Selvaraj

[Feng_C_]

Hi Surenthar,

Many thanks for your reply.

I still have two questions.

1) I found one of your team members replied a researcher on SGX memory limited (https://software.intel.com/en-us/forums/intel-software-guard-extensions-intel-sgx/topic/607004 ;). According to Simon, the 90MB memory limited is due to the hardware constraint. Currently, all of the hardware only support V1 instruction, which does not support `dynamic memory allocating. In addition, I also found your reply to another researcher on SGX 90 MB memory limited (https://software.intel.com/en-us/forums/intel-software-guard-extensions-intel-sgx/topic/670261 ), and you also mentioned the V2 instruction is unavailable currently.

So, I just wanna make sure if the V2 instruction is available for Linux platform now? Are all of the 4GB memory protected by hardware, or just emulator?

2) I found an issue in Linux SDK: Although I only set the StackMaxSize to 0x40000 (0.25 MB), I can allocate five 2GB arrays (totally 10 GB) in the stack and the program correctly under prerelease hardware mode. For the debug hardware mode, the program failed. In addition, I also tested the prerelease mode in Windows 10, the program also failed. So, is there any bug in Linux prerelease mode? 

Attached are key step screenshots for question 2. 

Our server CPU is Intel® Xeon® Processor E3-1275 v5.

Best,

Feng

[Surenthar_S_Intel]

Hi Feng,

#1, v2 instructions are a hardware feature, not an OS or software feature. No currently shipping Intel CPU supports the v2 instructions and we have not announced the product intercepts or projected timelines for their availability. I don’t understand the reference to an emulator in the question “Are all of the 4GB memory protected by hardware, or just emulator?”. Linux supports paging so if you specify a 4 GB HeapSize you get 4 GB of protected enclave memory, it just has to swap pages to access it.

#2 Thank for the report and we have forwarded it on to the SDK developers.

Thanks and Reagrds,
Surenthar Selvaraj

[Surenthar_S_Intel · Accepted solution]

Hi Feng,

The physical protected memory is limited to the PRMRR size set in BIOS and the max we support at this time is 128MB in Skylake. The reason why you are able to set the heapsize you set is because of the paging support in Linux driver and we don't have this support in Windows at this time. Similar to how memory is managed in OS, enclave pages are managed similarly.

Thanks and Reagrds,
Surenthar Selvaraj

[Feng_C_]

Hi Surenthar,

Many thanks for your reply. For the "emulator", I meant the program emulates a compromised mechanism to allocate 4 GB memory, which may not be as safe as the 90 MB secure memory in Windows. However, based on your reply, I got known it is the paging mechanism: to access more than 128 MB, the driver can swap the data between the protected memory and the paging area, and implements the encryption and decryption, right?  

For #2, it turns out the unused part of10 GB memory is optimized out by the compiler in Linux prerelease mode,  but these memories will not be optimized out in Windows prerelease mode or Linux Debug mode. 

Best,

Feng 

[Surenthar_S_Intel]

Feng C. wrote:

Hi Surenthar,

Many thanks for your reply. For the "emulator", I meant the program emulates a compromised mechanism to allocate 4 GB memory, which may not be as safe as the 90 MB secure memory in Windows. However, based on your reply, I got known it is the paging mechanism: to access more than 128 MB, the driver can swap the data between the protected memory and the paging area, and implements the encryption and decryption, right?  

For #2, it turns out the unused part of10 GB memory is optimized out by the compiler in Linux prerelease mode,  but these memories will not be optimized out in Windows prerelease mode or Linux Debug mode. 

Best,

Feng 

Hi Feng,

Yes, it will all be protected memory on Linux. The Linux driver supports paging, which allows the enclave to access the additional RAM via page swaps. The Windows kernel does not yet support this feature, so it is limited to ~90 MB.

-Surenthar Selvaraj

[Feng_C_]

Hi Surenthar,

Thank you for you reply.

I have implemented some simple C++ programs (random access the allocated array) both in Linux and windows (both systems run the same hardware), and the Linux program generally 10+ times slower than the Windows program even if the maximum memory is set less than 90 MB. So, I guess the paging mechanism is enabled by default,  Just wanna know if there is any method that I can disable Linux paging in order that the programs can only run in the EPC memory?

Best,

Feng

[Surenthar_S_Intel]

Hi Feng,

Paging is enabled by default on Linux and there isn’t any option to disable it.

Share your copy of program with us (both linux and windows) will look and revert back to you

-Surenthar

[Fan]

Hi, Feng,

I'm very interested in the performance overhead of paging. Did you finally figure out the reason for 10x slower? 

Fan

Feng C. wrote:

Hi Surenthar,

Thank you for you reply.

I have implemented some simple C++ programs (random access the allocated array) both in Linux and windows (both systems run the same hardware), and the Linux program generally 10+ times slower than the Windows program even if the maximum memory is set less than 90 MB. So, I guess the paging mechanism is enabled by default,  Just wanna know if there is any method that I can disable Linux paging in order that the programs can only run in the EPC memory?

Best,

Feng

[Rodolfo_S_]

Hi, Fan.

Some people have run experiments to evaluate the overhead of using SGX enclaves compared to the standard C library (without any data protection mechanism), due to memory swap.

The results show that randomly accessing memory (memory reads) in SGX enclaves can be up to 350x slower than doing it outside enclaves, using standard C library. The picture below show the full evaluation of the SGX performance.

Best regards,

Rodolfo

Fabian N. wrote:

Hi, Feng,

I'm very interested in the performance overhead of paging. Did you finally figure out the reason for 10x slower? 

Fan

[Benny_F_]

Hello Surenthar,

is there any update regarding paging support for Windows? Will it be available in the foreseeable future?

Best regards
Benny

[Meysam_t_]

Hello everyone,

I have a question about EPC in SGX and why it is not possible to increase the size of EPC more than 128MB? Because if we do so, enclave does not need to suffer from bringing pages into EPC or taking pages out of EPC so frequently, which has a significant overhead?

[Garg__Shivam]

Selvaraj, Surenthar (Intel) wrote:

Hi Feng,

Paging is enabled by default on Linux and there isn’t any option to disable it.

Share your copy of program with us (both linux and windows) will look and revert back to you

-Surenthar

Hi Selvaraj,

Can you share your thought why we are saying that we can't disable the paging from linux.

I referred the below page to disable to pagging : https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux

right now i am checking the SGX capability in term of size and even after disabling the paging  am able to allocate 400 MB memory on stack.