topic Quote:Selvaraj, Surenthar in Intel® Software Guard Extensions (Intel® SGX)

SGX protected memory limit in SGX

Feng_C_ — Thu, 21 Jul 2016 07:25:09 GMT

The SGX SDK reference indicated the SGX has limited protected memory as 128 MB. But I just found the memory limit can be extended almost up to 4GB in Linux platform (I set the HeapMaxSize = 0xF0000000 = 3.75GB). In addition, I allocated five 32 MB buffer array in the program (total 160 MB). The program run successfully.

So, I just wanna make sure if all of the 4GB are protected memory?

Thanks.

Feng

Hi Feng,

Surenthar_S_Intel — Fri, 22 Jul 2016 03:07:45 GMT

Hi Feng,

The limit of 128 MB comes from the BIOS. To go beyond that, there needs to be paging support. The Linux driver supports paging, but Windows does not.

Thanks and Reagrds,
Surenthar Selvaraj

Hi Surenthar,

Feng_C_ — Sat, 23 Jul 2016 07:46:33 GMT

Hi Surenthar,

Many thanks for your reply.

I still have two questions.

1) I found one of your team members replied a researcher on SGX memory limited (https://software.intel.com/en-us/forums/intel-software-guard-extensions-intel-sgx/topic/607004 ;). According to Simon, the 90MB memory limited is due to the hardware constraint. Currently, all of the hardware only support V1 instruction, which does not support `dynamic memory allocating. In addition, I also found your reply to another researcher on SGX 90 MB memory limited (https://software.intel.com/en-us/forums/intel-software-guard-extensions-intel-sgx/topic/670261 ), and you also mentioned the V2 instruction is unavailable currently.

So, I just wanna make sure if the V2 instruction is available for Linux platform now? Are all of the 4GB memory protected by hardware, or just emulator?

2) I found an issue in Linux SDK: Although I only set the StackMaxSize to 0x40000 (0.25 MB), I can allocate five 2GB arrays (totally 10 GB) in the stack and the program correctly under prerelease hardware mode. For the debug hardware mode, the program failed. In addition, I also tested the prerelease mode in Windows 10, the program also failed. So, is there any bug in Linux prerelease mode?

Attached are key step screenshots for question 2.

Our server CPU is Intel® Xeon® Processor E3-1275 v5.

Best,

Feng

Hi Feng,

Surenthar_S_Intel — Tue, 26 Jul 2016 06:17:42 GMT

Hi Feng,

#1, v2 instructions are a hardware feature, not an OS or software feature. No currently shipping Intel CPU supports the v2 instructions and we have not announced the product intercepts or projected timelines for their availability. I don’t understand the reference to an emulator in the question “Are all of the 4GB memory protected by hardware, or just emulator?”. Linux supports paging so if you specify a 4 GB HeapSize you get 4 GB of protected enclave memory, it just has to swap pages to access it.

#2 Thank for the report and we have forwarded it on to the SDK developers.

Thanks and Reagrds,
Surenthar Selvaraj

Hi Feng,

Surenthar_S_Intel — Tue, 26 Jul 2016 06:36:29 GMT

Hi Feng,

The physical protected memory is limited to the PRMRR size set in BIOS and the max we support at this time is 128MB in Skylake. The reason why you are able to set the heapsize you set is because of the paging support in Linux driver and we don't have this support in Windows at this time. Similar to how memory is managed in OS, enclave pages are managed similarly.

Thanks and Reagrds,
Surenthar Selvaraj

Hi Surenthar,

Feng_C_ — Tue, 26 Jul 2016 18:39:00 GMT

Hi Surenthar,

Many thanks for your reply. For the "emulator", I meant the program emulates a compromised mechanism to allocate 4 GB memory, which may not be as safe as the 90 MB secure memory in Windows. However, based on your reply, I got known it is the paging mechanism: to access more than 128 MB, the driver can swap the data between the protected memory and the paging area, and implements the encryption and decryption, right?

For #2, it turns out the unused part of10 GB memory is optimized out by the compiler in Linux prerelease mode, but these memories will not be optimized out in Windows prerelease mode or Linux Debug mode.

Best,

Feng

Quote:Feng C. wrote:

Surenthar_S_Intel — Thu, 28 Jul 2016 05:59:08 GMT

Feng C. wrote:

Hi Surenthar,

Many thanks for your reply. For the "emulator", I meant the program emulates a compromised mechanism to allocate 4 GB memory, which may not be as safe as the 90 MB secure memory in Windows. However, based on your reply, I got known it is the paging mechanism: to access more than 128 MB, the driver can swap the data between the protected memory and the paging area, and implements the encryption and decryption, right?

For #2, it turns out the unused part of10 GB memory is optimized out by the compiler in Linux prerelease mode, but these memories will not be optimized out in Windows prerelease mode or Linux Debug mode.

Best,

Feng

Hi Feng,

Yes, it will all be protected memory on Linux. The Linux driver supports paging, which allows the enclave to access the additional RAM via page swaps. The Windows kernel does not yet support this feature, so it is limited to ~90 MB.

-Surenthar Selvaraj

Hi Surenthar,

Feng_C_ — Thu, 28 Jul 2016 06:12:44 GMT

Hi Surenthar,

Thank you for you reply.

I have implemented some simple C++ programs (random access the allocated array) both in Linux and windows (both systems run the same hardware), and the Linux program generally 10+ times slower than the Windows program even if the maximum memory is set less than 90 MB. So, I guess the paging mechanism is enabled by default, Just wanna know if there is any method that I can disable Linux paging in order that the programs can only run in the EPC memory?

Best,

Feng

Hi Feng,

Surenthar_S_Intel — Wed, 14 Sep 2016 05:03:48 GMT

Hi Feng,

Paging is enabled by default on Linux and there isn’t any option to disable it.

Share your copy of program with us (both linux and windows) will look and revert back to you

-Surenthar

Hi, Feng,

Fan — Tue, 29 Nov 2016 05:00:43 GMT

Hi, Feng,

I'm very interested in the performance overhead of paging. Did you finally figure out the reason for 10x slower?

Fan

Feng C. wrote:

Hi Surenthar,

Thank you for you reply.

I have implemented some simple C++ programs (random access the allocated array) both in Linux and windows (both systems run the same hardware), and the Linux program generally 10+ times slower than the Windows program even if the maximum memory is set less than 90 MB. So, I guess the paging mechanism is enabled by default, Just wanna know if there is any method that I can disable Linux paging in order that the programs can only run in the EPC memory?

Best,

Feng

Hi, Fan.

Rodolfo_S_ — Tue, 06 Dec 2016 18:39:26 GMT

Hi, Fan.

Some people have run experiments to evaluate the overhead of using SGX enclaves compared to the standard C library (without any data protection mechanism), due to memory swap.

The results show that randomly accessing memory (memory reads) in SGX enclaves can be up to 350x slower than doing it outside enclaves, using standard C library. The picture below show the full evaluation of the SGX performance.

Best regards,

Rodolfo

Fabian N. wrote:

Hi, Feng,

I'm very interested in the performance overhead of paging. Did you finally figure out the reason for 10x slower?

Fan

Hello Surenthar,

Benny_F_ — Wed, 12 Apr 2017 11:33:27 GMT

Hello Surenthar,

is there any update regarding paging support for Windows? Will it be available in the foreseeable future?

Best regards
Benny

Hello everyone,

Meysam_t_ — Sat, 17 Jun 2017 00:11:16 GMT

Hello everyone,

I have a question about EPC in SGX and why it is not possible to increase the size of EPC more than 128MB? Because if we do so, enclave does not need to suffer from bringing pages into EPC or taking pages out of EPC so frequently, which has a significant overhead?

Quote:Selvaraj, Surenthar

Garg__Shivam — Thu, 12 Apr 2018 10:39:37 GMT

Selvaraj, Surenthar (Intel) wrote:

Hi Feng,

Paging is enabled by default on Linux and there isn’t any option to disable it.

Share your copy of program with us (both linux and windows) will look and revert back to you

-Surenthar

Hi Selvaraj,

Can you share your thought why we are saying that we can't disable the paging from linux.

I referred the below page to disable to pagging : https://serverfault.com/questions/684771/best-way-to-disable-swap-in-linux

right now i am checking the SGX capability in term of size and even after disabling the paging am able to allocate 400 MB memory on stack.