SGX Enclave systemetically crashing after some PUT operations

aghia · ‎11-30-2023

Hi,
I am using Intel SGX to build an in-memory confidential Key-Value Store with GET/PUT API.
I am using YCSB Benchmark to test its performance.
With Intensive GET operations, the application works just fine. But when I use PUT operations with secrets of 20 bytes for each, the enclave always crashes when I reach 7387 pushed secrets with the error "SGX_ERROR_ENCLAVE_CRASHED".

I am working on an Azure VM with Ubuntu 20.0.4 (Linux kernel 5) and SGX SDK 2.19.

This is my xml config file:

<MiscMask>0xFFFFFFFF</MiscMask>

</EnclaveConfiguration>

I tried to increase my HeapMaxSize, it worked for some time until pushing arround 14000 secrets, then it crashed. In next executions, it can store less secrets (5000 or less), that is quiet an inconsitent behavior.

The main issue I am facing is that EPC pages' swapping seems disabled since I should be able to push far way more secrets. It would slow down performance but would not make the enclave crash.

The other issue I am facing is that EDMM seems disabled. It would allow me to dynamically allocate EPC pages without having to set a number for HeapMaxSize, right ?
I have found that EDMM is only enabled for Linux kernel 6 or more, is it right ?

I hope you could point me to some solution for my problem.

Thank you in advance

Zulkifli_Intel · ‎12-01-2023

Hi aghia,

Thank you for reaching out to us.

We are checking this out and will update you as soon as possible.

Regards,

Zulkifli

Aznie_Intel · ‎12-04-2023

Hi Aghia,

That’s correct that SGX EDMM support was not added to the Linux Kernel until v6.0. You should upgrade to the v6.0 kernel or later. Here is the reference for EDMM, "SGX EDMM support (v6.0 or later)" at https://github.com/intel/sgx-emm.

For EDMM support, you need to Build and install the kernel with EDMM support by following the instructions in GitKernelBuild. You can use cpuid | grep SGX2 command to confirm whether your platform supports EDMM.

On another note, the SGX_ERROR_ENCLAVE_CRASHED error may indicate the SGX enclaves run under heavy memory pressure. In Linux, there is sgx_emmt provided by sgx-gdb. You may search for the chapter "Enclave Memory Measurement Tool" in the Developer Reference for more details. The tool helps to measure the usage of protected memory by enclave at runtime.

You may also refer to this Enclave Memory Measurement Tool for Intel® Software Guard Extensions (Intel® SGX) Enclaves.

Hope this helps.

Regards,

Aznie

aghia · ‎12-05-2023

Thank you for your answer.

I indeed used sgx_emmt in sgx-gdb and this is the result I got, just after the enclave crashing when setting <HeapMaxSize>0x100000</HeapMaxSize>(1MB):

[Peak stack used]: 13 KB
[Peak heap used]: 972 KB
[Peak reserved memory used]: 0 KB

And this is the result I got when setting <HeapMaxSize>0x1000000</HeapMaxSize> (16MB) just before crashing:
[Peak stack used]: 13 KB
[Peak heap used]: 1160 KB
[Peak reserved memory used]: 0 KB

I almost store the same amount secrets for both settings before enclave crashing while HeapMaxSize is way higher in the second case. It seems ineffective to increase HeapMaxSize in my case to improve enclave storage.
Why such a behaviour ?

Also, why do you mean by memory pressure ? Is it lot of writes in short period of time ? If so, would it mean that increasing HeapMaxSize wouldn't help to tackle this issue ?

Aznie_Intel · ‎12-27-2023

Hi Aghia,

Thank you for your patience.

It looks like there is a memory leak in your code. You can try to check the general memory usage of your application until it crashes to find the leak. It could also be that the memory limit of what the VM provides is reached.

On another note, there are Azure VMs with Ubuntu 22.04

Regards,

Aznie