topic Re:malloc(4096) failed in Intel® Software Guard Extensions (Intel® SGX)

malloc(4096) failed

0MagicDecoy0 — Thu, 28 Apr 2022 11:01:43 GMT

enclave configuration:

after I ran malloc(4096) several times, it failed to allocate a new memory location (returned a nullptr).

I'm using an SGX2-enabled cloud server (Alibaba cloud ECS g7t) which provides up to 4 GiB for enclave. Based on my understanding, this malloc will trigger the Enclave Dynamic Memory Management (EDMM) mechanism and assign a new page to the enclave after which this operation can be redone successfully.

Anyone can help me out? A code example of how to do SGX2-only EDMM would be great.

Re:malloc(4096) failed

Sahira_Intel — Thu, 28 Apr 2022 17:33:31 GMT

Hi,

What OS are you running on? Can you please send over your code including the enclave configuration file and the edl file.

Sincerely,

Sahira

Re: Re:malloc(4096) failed

0MagicDecoy0 — Fri, 29 Apr 2022 01:37:40 GMT

Hi Sahira,

I'm running on ubuntu 18.04 kernel version 4.15.0-175.

Here's my enclave config file:

<EnclaveConfiguration> <ProdID>0</ProdID> <ISVSVN>0</ISVSVN> <StackMinSize>0x2000</StackMinSize> <StackMaxSize>0x4000</StackMaxSize> <HeapMinSize>0x8000</HeapMinSize> <HeapInitSize>0x8000</HeapInitSize> <HeapMaxSize>0x50000000</HeapMaxSize> <TCSNum>1</TCSNum> <TCSMaxNum>10</TCSMaxNum> <TCSMinPool>1</TCSMinPool> <TCSPolicy>1</TCSPolicy> <DisableDebug>0</DisableDebug> <MiscSelect>0</MiscSelect> <MiscMask>0xFFFFFFFF</MiscMask> <ReservedMemInitSize>0x800000</ReservedMemInitSize> <ReservedMemMinSize>0x4000</ReservedMemMinSize> <ReservedMemMaxSize>0x50000000</ReservedMemMaxSize> </EnclaveConfiguration>

and here's my edl file:

enclave { trusted { public void ecall_hello_from_enclave([out, size=len] char* buf, size_t len, [in, size=8] int *allocBytes); }; };

in which buf is the output buffer for messages from the enclave and allocBytes is the number of bytes to be allocated (currently not used).

In my application code, ecall_hello_from_enclave function is called inside a loop & timed along side with ecall overhead

int cnt = 1; allocNum = 4096; while (cnt <= 200) { auto beginT = std::chrono::high_resolution_clock::now(); ecall_hello_from_enclave(global_eid, buffer, max_buf_len, &allocNum); auto endT = std::chrono::high_resolution_clock::now(); auto duration = std::chrono::duration_cast<std::chrono::microseconds>(endT - beginT); ++cnt; printf("\n\n\n Current Iteration: %d, Current Time: %s Enclave Output: %s\n Time Used: %.3f microseconds\n", cnt, asctime(get_time()), buffer, ((float) duration.count())); fflush(stdout); }

enclave code:

void ecall_hello_from_enclave(char *buf, size_t len, int *heapSize) { size_t size = len; char *tmpPtr = (char *) malloc(4096); char strBuf[100]; if (tmpPtr == nullptr) { size = snprintf(strBuf, 100, "Malloc failed!"); } else { size = snprintf(strBuf, 100, "Executed 4kBytes malloc operations"); } memcpy(buf, strBuf, size - 1); buf[size - 1] = '\0'; }

Hope this clarifies my problem.

Warmest regards,

Decoy

Re: Re:malloc(4096) failed

0MagicDecoy0 — Fri, 29 Apr 2022 02:54:41 GMT

Just changed my configuration of HeapMinSize, it comes to my attention that the initialized heap size after enclave initialization is the amount of memory set by HeapInitSize instead of HeapMinSize, which indicates that I am actually using SGX1 instead of SGX2 based on the developer's reference:

When an enclave created with the Linux* 2.0 SDK is executing on an Intel®
SGX 2.0 platform that is running the Intel® SGX 2.0 PSW, HeapMinSize is the
amount of heap available once the enclave completes initialization.

...

When an enclave created with the Linux* 2.0 SDK is executing on an older
Intel SGX platform or a platform running a previous version of PSW, the values
are interpreted differently. In this case HeapInitSize is the only relevant
field and it indicates the total amount of heap available to an enclave.

Am I using the wrong version of SGX? If so, how to install SGX 2.0 properly?

Re:malloc(4096) failed

JesusG_Intel — Mon, 02 May 2022 16:16:39 GMT

Hello 0MagicDecoy0,

We have a couple of articles that will help you. If your system supports SGX 2, then you just need to configure the enclave configuration file correctly and use malloc/free as you normally would.

Which Platforms Support Intel® Software Guard Extensions (Intel® SGX) SGX2?

Open a terminal and run: $ cpuid | grep -i sgx
Look for output: SGX2 supported = True

How Do I Create an Enclave with Intel® Software Guard Extensions 2 (Intel® SGX 2) Features to Allocate Memory Dynamically?

Sincerely,

Jesus G.

Intel Customer Support

Re: Re:malloc(4096) failed

0MagicDecoy0 — Mon, 02 May 2022 16:25:52 GMT

I'm pretty sure about the fact that my system supports SGX2 and I have double-checked with your prementioned method. Also, the enclave config file should be correct.

My concern is, is there any restriction on sgx-driver & sgx-sdk to make use of SGX2 features? Any specific branch to use when trying to build from source?

Re:malloc(4096) failed

JesusG_Intel — Mon, 02 May 2022 16:51:01 GMT

Hello 0MagicDecoy0,

There is no restriction on the SGX driver or SDK for SGX2. If your processor supports SGX2, then the SGX software will automatically detect it and benefit from the dynamic memory management functions.

Sincerely,

Jesus G.

Intel Customer Support

Re: Re:malloc(4096) failed

0MagicDecoy0 — Tue, 03 May 2022 10:38:42 GMT

Hi,

How can I verify whether it's using dynamic management functions or not?

I've tried to compare the enclave initialize time given different HeapMinSize but it showed no difference in loading time between the enclave with only 4kb HeapMinSize and the enclave with 3GB HeapMinSize (HeapMaxSize both set to 4GB).

Re: malloc(4096) failed

JesusG_Intel — Tue, 03 May 2022 22:54:21 GMT

Hello 0MagicDecoy0,

I was able to run your code successfully after I removed the HeapInitSize from the enclave config file.

If you set HeapInitSize, then you can only use that amount of memory, which is 0x8000 in your case.

So, to use SGX 2 dynamic memory allocation, set only HeapMinSize and HeapMaxSize, and remove HeapInitSize.

Sincerely,

Jesus G.

Intel Customer Support

Re: malloc(4096) failed

0MagicDecoy0 — Wed, 04 May 2022 01:55:33 GMT

Hi Jesus

Thanks for being so helpful! It indeed solved the question.

But there's one thing still not clear to me: how can I check whether the memory allocation is dynamic or static, i.e. is the enclave initialized with HeapMaxSize or HeapMinSize? As I don't know how to check the size of heap memory available right after enclave initialization (sgx_emmt only shows peak memory usage), I'm thinking about verifying it by measuring the enclave initialization time.

Based on my understanding, in SGX2, the enclave heap should be initialized with HeapMinSize memory at first and then the memory manager will dynamically allocate pages into this enclave as heap as requested during runtime, which implies that an enclave with EDMM should spend less time on enclave initialization since only HeapMinSize is required to be allocated.

However, I tried several settings with only HeapMinSize being different, but the time cost for sgx_create_enclave is roughly the same. To be more specific, with the HeapMaxSize setting at 3GB all the time, the enclave initialization time for HeapMinSize at 4kb is ~6.1 seconds, and the one for HeapMinSize at 2GB or 64MB or 500MB is roughly around that also. Is this a desirable behavior of an enclave with EDMM?

Looking forward to hearing from you.

Regards,
Decoy

Re:malloc(4096) failed

JesusG_Intel — Fri, 06 May 2022 16:54:25 GMT

Hello 0MagicDecoy0,

I apologize that I gave you wrong information earlier. There is a restriction on the SGX driver SGX 2 support. Only the old out-of-tree driver supports SGX 2. You can get this driver 2 ways:

Install the binary: sgx_linux_x64_driver_2.11.054c9c4c.bin

chmod 777 sgx_linux_x64_driver_2.11.054c9c4c.bin

sudo ./sgx_linux_x64_driver_2.11.054c9c4c.bin

build from source: https://github.com/intel/linux-sgx-driver

Please send us the output from

$ ls /dev/*sgx*

Sincerely,

Jesus G.

Intel Customer Support

Re: Re:malloc(4096) failed

0MagicDecoy0 — Sat, 07 May 2022 02:19:28 GMT

[root@Server ~]# ls /dev/*sgx*
/dev/sgx_enclave /dev/sgx_provision

/dev/sgx:
enclave provision

Re:malloc(4096) failed

JesusG_Intel — Mon, 09 May 2022 18:31:06 GMT

Hello 0MagicDecoy0,

It seems that you have installed the DCAP driver so are not taking advantage of SGX 2 features.

If you were running the OOT driver you would see output:

/dev/isgx

and

$ lsmod | grep -i sgx

isgx 53248 2

If you want to take advantage of SGX 2 EDMM features then you must uninstall the current driver and install the OOT driver.

First uninstall the current driver

sudo /opt/intel/sgxdriver/uninstall.sh # The path to uninstall.sh may differ for your system.

If the uninstall.sh script is missing, uninstall as follows:

sudo service aesmd stop

sudo rm -f $(find /lib/modules -name intel_sgx.ko)

sudo /sbin/depmod

sudo sed -i '/^intel_sgx$/d' /etc/modules

sudo rm -f /etc/sysconfig/modules/intel_sgx.modules

sudo rm -f /etc/modules-load.d/intel_sgx.conf

2. Install the OOT driver

Install the binary: sgx_linux_x64_driver_2.11.054c9c4c.bin

chmod 777 sgx_linux_x64_driver_2.11.054c9c4c.bin

sudo ./sgx_linux_x64_driver_2.11.054c9c4c.bin

build from source: https://github.com/intel/linux-sgx-driver

Sincerely,

Jesus G.

Intel Customer Support

Re: Re:malloc(4096) failed

0MagicDecoy0 — Tue, 10 May 2022 00:13:53 GMT

Thanks a lot!

It is working perfectly now!

Re:malloc(4096) failed

JesusG_Intel — Tue, 10 May 2022 20:05:55 GMT

This thread has been marked as answered and Intel will no longer monitor this thread. If you want a response from Intel in a follow-up question, please open a new thread.

Re: malloc(4096) failed

yunkaibai — Mon, 25 Jul 2022 12:17:35 GMT

I meet the similar questions.

According to the solutions, I reinstalled sgx driver but the enclave initialization time is still detetmined by HeapMaxSize.

The sgx driver is "isgx"

lsmod | grep -i sgx

isgx 57344 0

Encalve configuration:

The output of make:

tcs_num 10, tcs_max_num 10, tcs_min_pool 1
The required memory is 1076772864B.
The required memory is 0x402e4000, 1051536 KB.

Succeed.

I'm using an SGX2-enabled machine "Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz".

cpuid | grep -i sgx

SGX: Software Guard Extensions supported = true
SGX_LC: SGX launch config supported = true
Software Guard Extensions (SGX) capability (0x12/0):
SGX1 supported = true
SGX2 supported = true
SGX ENCLV E*VIRTCHILD, ESETCONTEXT = false
SGX ENCLS ETRACKC, ERDINFO, ELDBC, ELDUC = false
SGX attributes (0x12/1):
SGX EPC enumeration (0x12/n):

Anyone can help me out?

Re: malloc(4096) failed

yunkaibai — Mon, 25 Jul 2022 16:48:23 GMT

Besides, the kernel headers is 5.15.0-43-generic

uname -r

5.15.0-43-generic

I noticed that there are three different drives: In-kernel Driver, DCAP driver, OOT driver.

But The platform must be configured with only one of these drivers.

From kernel 5.11, the kernel supports in-kernel SGX driver.

So, I uninstalled the OOT driver(/dev/isgx).

Do I need to reinstall the in-kernel driver and how to reinstall it?