Complete procedure of moving an enclave thread to another physical core and side effects?

[Nelly]

Not a lot of resources and easy-to-get solutions can be found on the internet so I decided to ask this question here. Replied by an engineer from Intel is optimised however community members are also welcomed.

My question is for this feature (SGX) that assume I have an enclave program running on a hyper-threading enabled CPU (e.g my 8700k) within a random thread (logical core), that what is/are the complete procedure or steps if I want to either legitimately or illegally moving an enclave thread to another physical core? During that procedure/steps, is there any exception or interruption (e.g AEX) triggering to the enclave program or to the OS? Or even hardwares? One further question, will the thread that co-located with this core able to acknowledge that his "neighbour" has been moved away?

[JesusG_Intel]

Hello Nelly,

You are right, this info is difficult to find. There are a couple of ways of addressing this depending on exactly what you are trying to do.

• From a hardware/OS perspective:

There’s nothing special required to “move” an enclave from one core to another outside the normal way of moving any application between cores. If or when an OS scheduler wants to move or context switch out a process, all it must do is move/swap all the processor registers (ie. RAX, RBX, RIP, CR3, RSP, etc.) from one core to the other and then start execution back on the new core at the newly supplied RIP. There is nothing special that needs to happen simply because an enclave is part of an untrusted application.

There is no notification/interrupt to/from the enclave when this happens. Again, this is just like a normal context switch. There may be notifications within the OS kernel of this, but I have no idea.

• This thread gives recommendations on how a programmer can code their own OCALL or modify the SGX untrusted runtime to set thread affinity.

Sincerely,

Jesus G.

Intel Customer Support

[Nelly]

Hi Jesus

Thanks for your reply. I only got to see this thread back and just realise that the first link isn't available anymore.

Beside that, from what I've learnt regarding the operating system, in order to context switch out a process, (at least) an system interrupt will be issued from the operating system to the process. And that's how processor registers including those you mentioned will be stored in the main memory and then when it's resumed on another processor, it can be loaded to continue. I suppose that would happen to the enclave process as well since it's running as a user process at ring 3. Can you confirm this or anything I understood isn't correct?

Regards

[JesusG_Intel]

Hello Nelly,

You are correct. I would like to clarify something. Enclaves do not run in their own process. The enclave code gets compiled as a static library with the untrusted application. Enclaves run as separate threads within the process of the untrusted application. These diagrams illustrate that.

Sincerely,

Jesus G.

Intel Customer Support

[Nelly]

Hello Jesus

Thanks for your reply, I see where we made a misunderstanding. When talking about threads, I'm referring it to the two logical core that was split from one physical core. I'll use logical core for future discussion.I understand that enclave doesn't have a specific process but rather wrapping in the untrusted application as a whole. 

To clarify my question, what I'm asking is when this whole enclave process - that has the untrusted application wrapping it - get switched out from its original logical core and then scheduled to another core, how does the process works out?

By the way I feel like I've got the answer but just to confirm. This logical core switching process will surely lead to an AEX to the enclave as it will receive system interrupt for stopping its current execution, is that right? Also I have a further question that assume I have one enclave application that precisely occupied the two logical core of one physical core on the CPU, when a system interrupt only happened in one of the logical core occupied by the enclave application, does the whole enclave application perform an AEX or only the logical core that receive the interrupt will do that?

Regards

[JesusG_Intel · Accepted solution]

Hello Nelly,

Hello Nelly,

Enclaves execute within a particular thread as defined by the TCS, which executes on a logical processor core. All instructions occur within the context of a TCS, and there can be multiple TCS per application. The AEX instruction is issued to the particular logical core on which the enclave is running. All of this is explained in chapter 6 of Innovative Instructions and Software Model for Isolated Execution.

'The EENTER instruction is the method to enter the enclave under program control. To execute EENTER, software must supply an address of a TCS that is part of the enclave to be entered. The TCS indicates the location inside the enclave to transfer control and where inside the enclave AEX should store the register state. When a logical processor enters an enclave, the TCS is considered busy until the logical processors exits the enclave. SGX allows an enclave builder to define multiple TCS structures, thereby providing support for multithreaded enclaves…Resuming Execution after AEX: After system software has serviced the event that caused the logical process to exit an enclave, the logical processor can re-start execution using ERESUME."

Sincerely,

Jesus G.

Intel Customer Support

[JesusG_Intel]

This thread has been marked as answered and Intel will no longer monitor this thread. If you want a response from Intel in a follow-up question, please open a new thread.