How to count EPC paging, EPC hits and EPC misses?

[kai__chi]

Hi,

I am looking for a way to count EPC hits and misses, and EPC paging.

For EPC paging I thought of counting the invocations of the EWB instruction. For that I placed a counter in:

static void sgx_evict_page(struct sgx_encl_page *entry,
			   struct sgx_encl *encl)

Every invocation would equal to 4kB of data. Is that correct?

I actually tried it out and for an enclave that processed some data in 30s it showed me 6000MB of EPC paging. Could that really be the number?

I struggle to find a way to count EPC hits and misses. Is there a way to detect an EPC access?

Best,

Kajetan

[JananiC_Intel]

Hi,

Thanks for posting in Intel forums.

Could you share your platform details?

Regards,

Janani Chandran

[kai__chi]

I'm running Ubuntu 18.04.5 with kernel 5.3.0 on Intel Core i7-8565U CPU and  SGX driver v2.11

[JananiC_Intel]

Hi,

Sorry for the delay .We are checking with the concerned team. We will get back to you shortly.

Regards,

Janani Chandran

[kai__chi]

Hi Janani,

Any updates from the team?

[Vinutha_SV]

Hi,

We don't have any SGX-specific events.

If a system configuration provides for paging-out of EPC pages, then paging effects, such as page faults within ELRANGE of an enclave (after which the OS/VMM are supposed to page an EPC page in), should be trackable using TLB-Miss-related precise events: for instance, MEM_INST_RETIRED.STLB_MISS_LOADS_PS and MEM_INST_RETIRED.STLB_MISS_STORES_PS.

Each eviction with EWB is 4k, but 6GB of paging in 30 seconds looks suspicious. you can probably look into the correctness of handling EWB error codes and not counting unsuccessful operations.

you can collect MEM_INST_RETIRED.STLB_MISS_LOADS_PS and MEM_INST_RETIRED.STLB_MISS_STORES_PS events and compare if the number of TLB misses (as a rough estimate for page faults) is on par with the number of EPC evictions via EWB.

[kai__chi]

Hi Vinutha,

Many thanks for the detailed response.

I moved the counter to the EWB function:

static int __sgx_ewb(struct sgx_encl *encl,
		     struct sgx_encl_page *encl_page)

and it now looks like this:

ret = __ewb(&pginfo, epc,
            (void *)((unsigned long)va + encl_page->va_offset));
if (ret == SGX_SUCCESS) {
    counter++;
}

I am still getting the kind of large numbers I mentioned in the previous posts (6GB in 30s). The counter is set of course per enclave so every enclave initializes its counter to 0 at the creation time. Is there any error in my logic?

Meanwhile, I will also count the hardware events you have mentioned and compare the results, however, I would still like to make it work for the EWB instruction.

Looking forward to your response!

[kai__chi]

Hi!

Any updates on this issue?

best,

Kai

[Vinutha_SV]

Hi,

Sorry for the delayed response. As you are getting 6GB in 30s, only way to check the correctness is by using the events I mentioned earlier. Do you get similar number when you measure with the events?

[kai__chi]

Hi,

I am getting some confusing results and would be grateful for some help.

I create an enclave with 600MB heap. The last lines from the enclave compilation indicate the required memory:

tcs_num 9, tcs_max_num 9, tcs_min_pool 9
The required memory is 633090048B.
The required memory is 0x25bc3000, 618252 KB.
Succeed.

My program creates an enclave, executes an ECALL (for about 15s), and destroys the enclave.

I surround the ECALL with the measurements of EWB and the events you mentioned. I measure the events using PCM

The EWB counter before ECALL shows 284,638 (~1.08GB) and just after the ECALL 728,571 (~2.77GB). So that would mean around 1.7GB of EPC paging just during the ECALL.

The events counters are as follows:

[ 24.7683][  PCM] MEM_INST_RETIRED.STLB_MISS_LOADS : 6800364
[ 24.7683][  PCM] MEM_INST_RETIRED.STLB_MISS_STORES: 20741208

How can I interpret the results to know if my EWB counter is correct?

I am running Ubuntu 18.04.5 with kernel 5.3.0, SGXv1, SGX-SDK v2.13.3, CPU Intel Core i7-8565U.

[JesusG_Intel]

Hello chi__kai,

Your processor has only 128MB of EPC so it is possible that you will have a lot paging if you are trying to process large amounts of data.

How much data are you trying to process in the enclave?

Sincerely,

Jesus G.

Intel Customer Support

[JesusG_Intel]

Hello Chi__kai,

We have not heard from you in several days so we will no longer monitor this thread. We hope you were able to resolve your issue. Please start a new thread if you need further help.

Sincerely,

Jesus G.

Intel Customer Support

[kai__chi]

Hi Jesus,

It seems to me a bit rushed to close a topic after four days of inactivity (considering it once took the Intel team almost 12 weeks to reply, see messages between 04-21-2021 and 07-12-2021). Hope you can still take a look at this thread.

The processed data was only 600 MB. In the messages above I provided detailed results from hardware counters requested from your colleagues. Would you mind taking a look at them and help me in the interpretation?

Best,

Kai

[JesusG_Intel]

Hello chi__kai,

I apologize for the precipitous closure of this thread.

The amount of paging you are seeing is not inconceivable since your processor has only 128MB of EPC and your data is 600MB. It all depends on how you are accessing the data within your enclave. If you are doing searches on your data, you may incur a lot of paging by accessing the 600MB often.

Sincerely,

Jesus G.

Intel Customer Support

[kai__chi]

Hi Jesus,

Yes, I totally agree that it depends on the data access patterns. I would just like to confirm that the numbers are not off. Vinutha recommended to measure MEM_INST_RETIRED.STLB_MISS_LOADS and  MEM_INST_RETIRED.STLB_MISS_STORES, which I did but I don't know how to interpret the results. I would be grateful if Vinutha or you could help me with that.

Best,

Kai

[JesusG_Intel]

Hello kai__chi,

We are still investigating this issue. We will respond to this thread as soon as we have an update.

Sincerely,

Jesus G.

Intel Customer Support

[JesusG_Intel]

Hello chi__kai,

We are still investigating this issue. We apologize for the delay.

Sincerely,

Jesus G.

Intel Customer Support

[JesusG_Intel]

Hello Kai_chi,

I apologize for the long delay. Unfortunately, we have been unable to determine a method for counting EPC paging/hits/misses using VTune.

It seems that your estimate of 6GB is reasonable if you are traversing your array often.

Sincerely,

Jesus G.

Intel Customer Support

[JesusG_Intel]

Hello kai_chi,

Since we do not have a solution to your question, we will close this thread. I apologize for the long wait only to be told at the end that we do not have an answer. If you have other SGX-related questions, please open a new thread and we will be happy to help you.

Sincerely,

Jesus G.

Intel Customer Support

[kai__chi]

Hi Jesus,

Thanks a lot for your time and responses.

Thanks for confirming that VTune is not capable of doing it. I will try to get more information about the two TLB hardware events and see if I can use this data effectively.

Best,

Kai