set enclave thread affinity

[kai__chi]

I am looking for a way to assign a CPU core to an enclave thread. In a normal application I would use pthread_attr_setaffinity_np.

I understand that from inside the enclave I can't do much, but I can set TCSPolicy to 0 (Binding Mode) and I'll have each Trusted Thread Contexts bounded to an untrusted thread. Now the question is - can I assign a CPU core to an untrusted thread?

Maybe this thread could be helpful . There it says: "SGX thread is mapped directly to logical processor.". What does that really mean though? Does that imply affinity?

To try it out empirically, I ran a multi-threaded application and looked it up with vtune (see the picture below). It clearly schedules the thread on many logical cores. Is there a way to map it to only one?

Thanks!

[JesusG_Intel]

Hello chi_kai,

Please clarify further. Are you saying that you tried to use pthread_attr_setaffinity_np in your untrusted thread and it did not work to bind your trusted enclave to that thread?

If you use pthread_attr_setaffinity_np in an untrusted app before jumping into an enclave (ECALL), it will pin enclave execution to that same thread.

[kai__chi]

Hello Jesus,

Thanks for a quick response.

I have not tried to set affinity in uRTS. In the untrusted part, I don't do anything regarding creating the threads. In the config file, I set the TCSNum field. In the trusted part (inside of the ECALL), I call pthread_create from sgx_pthread. At this point I'm already inside the enclave and can't set threads' affinity.

What would be the other options? Can I create the threads in the untrusted part (where I can use the complete pthread library) and somehow bind them to the enclave? But then how would I assign the work to them inside the enclave?

[JesusG_Intel]

Hello chi_kai,

You answered your own question in your first post:

"In a normal application I would use pthread_attr_setaffinity_np. I understand that from inside the enclave I can't do much, but I can set TCSPolicy to 0 (Binding Mode) and I'll have each Trusted Thread Contexts bounded to an untrusted thread."

Use TCSPolicy 0 and pthread_attr_setaffinity_np in your untrusted thread before making the ecall to the enclave.

[kai__chi]

Thanks for your response.

I'm not sure how to apply your tips. Currently, I create the threads inside of the ECALL where I can't use the attr parameter (from the Developer Reference of pthread_create: "The attr is not supported inside the Enclave, so the new thread will be created with PTHREAD_CREATE_JOINABLE."). You advised to set the affinity before making the ECALL, but at this point the threads don't exist yet (they're created from inside of the enclave).

 

[JesusG_Intel]

Hello chi_kai,

If you set the untrusted SW thread's affinity using pthread_attr_setaffinity_np and set the TCSPolicy to 0 in your enclave configuration xml file, then calling into the enclave from the untrusted SW thread would stay on the same logical processor. This ensures your trusted enclave thread runs on the same logical processor as your untrusted thread.

This is what you want, right? You want to ensure sure that an enclave’s thread runs on the same logical processor as the untrusted thread.

[kai__chi]

Thanks, Jesus.
I want to do something a little bit different. I want to do an ECALL and inside of the enclave I want to create N threads and assure that each thread is assigned to one logical processor, so thread 0 executes on cpu-0, thread 1 on cpu-1, etc.
If you look at the screenshot from the original message you can see that the thread with TID 7795 executes on all processors from cpu-0 to cpu-7. (This is by the way a thread created inside of the enclave. It is not an untrusted thread.) What I am trying to achieve is that this process executes on one core only.
Hope it's clearer now.

[JesusG_Intel · Accepted solution]

Hello chi_kai, thank you for explaining. Engineering came back with a few suggestions. There is no easy way....

The best way to accomplish what chi_kai wants is to use the trusted Pthread library.

Unfortunately, we only support a subset of the Pthread API and pthread_attr_setaffinity_np is not one of those.

However, if chi_kai is willing to modify the uRTS, they can get it working. There are two options described below.

1. Using the trusted pthread library

The enclave calls pthread_create, which triggers an OCALL that calls pthread_create in the uRTS.

Chi_kai needs to modify the uRTS to have pthread_create_ocall call pthread_attr_setaffinity_np before the call to pthread_create.

Note that pthread_create_ocall needs to keep track of the cores this routine pins threads to using some data structure.

The routine could also call pthread_attr_getaffinity_np to ensure that no thread is pinned to the same core as the first thread that calls pthread_create.

I’m not aware of an API to enumerate all the threads running within a process. For that reason pthread_create_ocall will be unaware of threads pinned outside this routine, with the exception of the threads making the OCALL.

2. Define own thread_create OCALL

It’s also possible to define a trusted thread_create that operates similarly to pthread_create.

If needed, the new thread_create_ocall could take an extra parameter to pass the core the thread about to be created should be pinned to.

The advantage is that no changes to the uRTS are necessary.

On the other hand, chi_kai has to replicate the pthread_create and pthread_create_ocall functionality available in the trusted pthread library and uRTS, respectively.

[kai__chi]

Thank you very much, Jesus!

I hoped for a straightforward way but the ideas from the engineering team are definitely worth experimenting with!

[JesusG_Intel]

This thread has been marked as answered and Intel will no longer monitor this thread. If you want a response from Intel in a follow-up question, please open a new thread.