Can I dynamically change an SGX memory page's permission bits?

[Ko__Ronny]

Hi,

I have to implement an SGX software which has to dynamically change the memory location of the code section. If I do this in a regular C program, I need to allocate a new memory page, call mprotect() and set the page's PROT_EXEC bit to 1, copy the original code section to the newly allocated memory, and jump to here.
 
However, I do "#include <sys/mman.h>" which defines mprotect(), the SGX can't find this header. Does this mean SGX has no way to change the memory page permission bit of a dynamically allocated memory?

[Francisco_C_Intel]

Some processors support EMODPE and EMODPR instructions as described in the developer manual.

https://software.intel.com/sites/default/files/managed/48/88/329298-002.pdf

If you have such a processor, you can use sgx_virtual_protect in Windows, declared C:\Program Files (x86)\Intel\IntelSGXSDK\include\sgx_tedmm.h. There is an "SGX2Enclave" sample included in the Windows SDK.

In Linux, there is an SGX2 driver https://github.com/intel/linux-sgx-driver/tree/sgx2 and I believe the API is trts_mprotect - https://github.com/intel/linux-sgx/blob/d10cabebb5512878e84f5d21cdf27c39c428ffe2/sdk/trts/trts_ecall.cpp

Hope this helps.

Francisco

[Ko__Ronny]

Thanks very much for your reply. I tried to use trts_mprotect() as you suggested, but I get an "Illegal instruction" error when calling the ENCLU instruction from file sdk/trts/linux/trts_pic.S:

DECLARE_GLOBAL_FUNC do_emodpe
    SE_PROLOG
    mov     $SE_EMODPE, %eax
    ENCLU
    SE_EPILOG

Does this mean my computer's processor doesn't support ENCLU instruction? If so, is there absolutely no way for me to executable dynamic memory page from within the enclave? Would it be not possible even if I pass the enclave's dynamic memory page's address to the untrusted code (by using ocall) to ask for modifying the permission bit to PROT_EXEC?

Ronny

[Francisco_C_Intel]

It appears EMOD instructions are not supported on your system. You can look at the CPUID check for this:

  https://github.com/intel/linux-sgx/blob/master/psw/urts/linux/edmm_utility.cpp

  is_cpu_support_edmm()

The NUCs below should have processors that support SGX2 (EDMM) instructions.

https://ark.intel.com/content/www/us/en/ark/products/126135/intel-nuc-kit-nuc7cjyh.html
https://ark.intel.com/content/www/us/en/ark/products/126137/intel-nuc-kit-nuc7pjyh.html

If you do not have hardware support, then no, you cannot modify the permissions of a EPC page.

Thanks,

Francisco

[Ko__Ronny]

Thanks for further help. As you said, is_cpu_support_edmm() returns false on my computer.

However, I could manage to change the enclave's memory permission by implementing an ocall. I allocated a memory page in Enclave.cpp:

void * allocate_aligned(size_t size, size_t alignment)

{
  const size_t mask = alignment - 1;
  const uintptr_t mem = (uintptr_t) malloc(size + alignment);
  return (void *) ((mem + mask) & ~mask);
}

void ocall_create_executable_memory(char* s)
{  char* page = allocate_alligned(0x1000, 0x1000);
    ocall_pass_memory((size_t)page);
}

and I passed this memory page's address to App.cpp and let it change this page's permission:

void ocall_create_executable_memory(size_t s)
{
    if (mprotect((char*)s, 0x1000, PROT_NONE)) {
        perror("Couldn’t mprotect");
        exit(errno);
    }
}

I change the page permission to PROT_NONE from within App.cpp, after then Enclave.cpp cannot read this page (it exits with an error). So I think it works as expected. Do you see any logical flaws in my approach?

 

[Francisco_C_Intel]

mprotect will modify the OS permissions on that page , but not the Enclave Page Cache Map (EPCM) permissions. You need EMOD instructions to modify the EPCM permissions on that page.

[Ko__Ronny]

I understand that the OS can't change the memory permission of the Enclave, but if I run the code, it somehow works. I attached my code. You can run it and see: 

$ unzip sgx-mprotect.zip
$ cd sgx-mprotect
$ make
$ ./App_exe
is_cpu_support_edmm(): 0
pagesize: 4096 bytes, function1()'s size: 15 bytes, function2()'s size: 39 bytes

ocall_mprotect(): received memory address 0x7fc02185d000 from SGX
ocall_mprotect(): modified page permission of memory address 0x7fc02185d000 to 7
SGX's dynamic_execute(): copied function1() to memory address 0x7fc02185d000
SGX's dynamic_execute(): executed original function1() --> returned 1
SGX's dynamic_execute(): executed dynamically created function1()--> returned 1
SGX's dynamic_execute(): copied function1() to memory address 0x7fc02185d000
SGX's dynamic_execute(): executed original function2() --> returned 83
SGX's dynamic_execute(): executed dynamically created function2()--> returned 83
Done

However, if you comment out Enclave.cpp's line 30: "sgx_mprotect(mem_aligned, PAGE_SIZE, 7);" and run it again:

$ make
$ ./App_exe 
is_cpu_support_edmm(): 0
pagesize: 4096 bytes, function1()'s size: 15 bytes, function2()'s size: 39 bytes

SGX's dynamic_execute(): copied function1() to memory address 0x7f67e31d9000
SGX's dynamic_execute(): executed original function1() --> returned 1
Segmentation fault (core dumped)

Does this imply that the OS can change the page permission of the Enclave?


 

[Ko__Ronny]

I just realized that the memory address allocated by SGX's malloc() call can be also accessed by the OS. I thought malloc() within SGX is the Enclave's own memory, but is this actually the OS's memory space?

[Scott_R_Intel]

Hi Ronny.

It appears from your Makefile that you are running in SIM (simulation) mode (you've commented out "HW")...  can you confirm?  If so, then yes, the OS would be able to access this memory.  You will need to build and run in HW mode for the SGX protections to be used.

Regards.

Scott

[Ko__Ronny]

Yes, you are correct, it was my flaw that I compiled in SIM Mode. As I compile in HW mode, the OS can't change the SGX's page permission successfully anymore and the SGX trying to execute the memory page ends with a segmentation fault. Thanks a lot for finding out this bug for me.

I really need an executable memory page from within the Enclave... Just before the Enclave program starts, could it be possible to configure the Enclave to make it statically pre-allocate huge memory pages whose executable bit is 1? If this is ever possible, which (either configuration or source code) file should I modify to enforce this?

[Francisco_C_Intel · Accepted solution]

Here are some thoughts:

1. You can allocate a section (using pragmas) in your .so enclave file.

2. Then you can modify the permissions on that section to be executable.

3. After making those modifications, use the sgx signing tool to sign your enclave.

For 2, you web searches show some usages of objcopy, ld -N, or writing your own program that modifies them. I believe there's also a way to write it in assembly that ensures the permissions stay RWX.

SGX has to do this for its "Protected Code Loader" feature, but I am not familiar with its implementation, I think it is done by calling a program built with the first link below.

https://github.com/intel/linux-sgx/blob/master/sdk/encrypt_enclave/encryptip.cpp , function "update_flags"

The sample and the underlying library it uses are:

https://github.com/intel/linux-sgx/tree/master/SampleCode/SampleEnclavePCL

https://github.com/intel/linux-sgx/tree/master/sdk/protected_code_loader

Thanks,

Francisco

[Ko__Ronny]

Hi,

Thanks very much for your kind directions. As you hinted me, I allocated a huge char[] array as a global variable in my Enclave application C code and compiled it to create an enclave.so. Then I inserted the following code in the update_flag() function in the linux-sgx/sdk/encrypt_enclave/encryptip.cpp file:

      if (dat->phdr[segidx].p_type == 1 && dat->phdr[segidx].p_flags == (PF_W | PF_R))
          dat->phdr[segidx].p_flags = (Elf64_Word)(PF_W | PF_R | PF_X);

to change the page permission of enclave.so's ELF data section to be readable & writable & executable, and compiled this program by typing "make". Then I ran this compiled memory permission modifier program with the above enclave.so as an input to make its ELF data section executable, and then compiled it again (by typing "make") to re-sign the modified enclave.so. Now the finalized enclave can dynamically write code to its global char[] array and execute it properly. I attached my Enclave memory permission modifier program for anyone's reference, and the following is the output I get:

$ cd ~
$ unzip sgx-app.zip
$ mv elf_data_segment_mark_x.zip ~/linux-sgx/sdk
$ cd ~/linux-sgx/sdk
$ unzip elf_data_segment_mark_x.zip
$ cd elf_data_segment_mark_x
$ make
$ cd ~/sgx-app
$ make clean
$ make
$ ~/linux-sgx/sdk/elf-data-segment-mark-x/sgx_data_segment_mark_x -i enclave.so -o enclave.so
$ make   # This re-creates enclave.signed.so based on the modified enclave.so
$ ./App_exe

is_cpu_support_edmm(): 0
pagesize: 4096 bytes, function1()'s size: 15 bytes, function2()'s size: 39 bytes

ocall_mprotect(): received memory address 0x7ff379231000 from SGX
ocall_mprotect(): modified page permission of memory address 0x7ff379231000 to 7
SGX's dynamic_execute(): copied function1() to memory address 0x7ff379224960
SGX's dynamic_execute(): executed original function1() --> returned 1
SGX's dynamic_execute(): executed dynamically created function1()--> returned 1
SGX's dynamic_execute(): copied function1() to memory address 0x7ff379224960
SGX's dynamic_execute(): executed original function2() --> returned 83
SGX's dynamic_execute(): executed dynamically created function2()--> returned 83
Done

Thanks very much for all of your great support :)