Intel® Software Guard Extensions (Intel® SGX)
Discussion board focused on hardware-based isolation and memory encryption to provide extended code protection in solutions.

How to count EPC paging, EPC hits and EPC misses?

kai__chi
Novice
4,694 Views

Hi,

I am looking for a way to count EPC hits and misses, and EPC paging.

For EPC paging I thought of counting the invocations of the EWB instruction. For that I placed a counter in:

static void sgx_evict_page(struct sgx_encl_page *entry,
			   struct sgx_encl *encl)

Every invocation would equal to 4kB of data. Is that correct?

I actually tried it out and for an enclave that processed some data in 30s it showed me 6000MB of EPC paging. Could that really be the number?

 

I struggle to find a way to count EPC hits and misses. Is there a way to detect an EPC access?

 

Best,

Kajetan

Labels (3)
0 Kudos
19 Replies
JananiC_Intel
Moderator
4,631 Views

Hi,


Thanks for posting in Intel forums.


Could you share your platform details?


Regards,

Janani Chandran


0 Kudos
kai__chi
Novice
4,627 Views

I'm running Ubuntu 18.04.5 with kernel 5.3.0 on Intel Core i7-8565U CPU and  SGX driver v2.11

0 Kudos
JananiC_Intel
Moderator
4,582 Views

Hi,


Sorry for the delay .We are checking with the concerned team. We will get back to you shortly.


Regards,

Janani Chandran


0 Kudos
kai__chi
Novice
4,528 Views

Hi Janani,

Any updates from the team?

0 Kudos
Vinutha_SV
Moderator
4,445 Views

Hi,

We don't have any SGX-specific events.

If a system configuration provides for paging-out of EPC pages, then paging effects, such as page faults within ELRANGE of an enclave (after which the OS/VMM are supposed to page an EPC page in), should be trackable using TLB-Miss-related precise events: for instance, MEM_INST_RETIRED.STLB_MISS_LOADS_PS and MEM_INST_RETIRED.STLB_MISS_STORES_PS.


Each eviction with EWB is 4k, but 6GB of paging in 30 seconds looks suspicious. you can probably look into the correctness of handling EWB error codes and not counting unsuccessful operations.

 

you can collect MEM_INST_RETIRED.STLB_MISS_LOADS_PS and MEM_INST_RETIRED.STLB_MISS_STORES_PS events and compare if the number of TLB misses (as a rough estimate for page faults) is on par with the number of EPC evictions via EWB.


0 Kudos
kai__chi
Novice
4,422 Views

Hi Vinutha,

Many thanks for the detailed response.

I moved the counter to the EWB function:

static int __sgx_ewb(struct sgx_encl *encl,
		     struct sgx_encl_page *encl_page)

and it now looks like this:

ret = __ewb(&pginfo, epc,
            (void *)((unsigned long)va + encl_page->va_offset));
if (ret == SGX_SUCCESS) {
    counter++;
}

I am still getting the kind of large numbers I mentioned in the previous posts (6GB in 30s). The counter is set of course per enclave so every enclave initializes its counter to 0 at the creation time. Is there any error in my logic?

 

Meanwhile, I will also count the hardware events you have mentioned and compare the results, however, I would still like to make it work for the EWB instruction.

Looking forward to your response!

0 Kudos
kai__chi
Novice
4,390 Views

Hi!

Any updates on this issue?

best,

Kai

0 Kudos
Vinutha_SV
Moderator
4,279 Views

Hi,

Sorry for the delayed response. As you are getting 6GB in 30s, only way to check the correctness is by using the events I mentioned earlier. Do you get similar number when you measure with the events?


0 Kudos
kai__chi
Novice
4,256 Views

Hi,

I am getting some confusing results and would be grateful for some help.

I create an enclave with 600MB heap. The last lines from the enclave compilation indicate the required memory:

 

tcs_num 9, tcs_max_num 9, tcs_min_pool 9
The required memory is 633090048B.
The required memory is 0x25bc3000, 618252 KB.
Succeed.

 

 

My program creates an enclave, executes an ECALL (for about 15s), and destroys the enclave.

I surround the ECALL with the measurements of EWB and the events you mentioned. I measure the events using PCM

The EWB counter before ECALL shows 284,638 (~1.08GB) and just after the ECALL 728,571 (~2.77GB). So that would mean around 1.7GB of EPC paging just during the ECALL.

The events counters are as follows:

[ 24.7683][  PCM] MEM_INST_RETIRED.STLB_MISS_LOADS : 6800364
[ 24.7683][  PCM] MEM_INST_RETIRED.STLB_MISS_STORES: 20741208

 

How can I interpret the results to know if my EWB counter is correct?

I am running Ubuntu 18.04.5 with kernel 5.3.0, SGXv1, SGX-SDK v2.13.3, CPU Intel Core i7-8565U.

0 Kudos
JesusG_Intel
Moderator
4,161 Views

Hello chi__kai,


Your processor has only 128MB of EPC so it is possible that you will have a lot paging if you are trying to process large amounts of data.


How much data are you trying to process in the enclave?


Sincerely,

Jesus G.

Intel Customer Support


0 Kudos
JesusG_Intel
Moderator
4,129 Views

Hello Chi__kai,


We have not heard from you in several days so we will no longer monitor this thread. We hope you were able to resolve your issue. Please start a new thread if you need further help.


Sincerely,

Jesus G.

Intel Customer Support


0 Kudos
kai__chi
Novice
4,099 Views

Hi Jesus,

 

It seems to me a bit rushed to close a topic after four days of inactivity (considering it once took the Intel team almost 12 weeks to reply, see messages between 04-21-2021 and 07-12-2021). Hope you can still take a look at this thread.

 

The processed data was only 600 MB. In the messages above I provided detailed results from hardware counters requested from your colleagues. Would you mind taking a look at them and help me in the interpretation?

 

Best,

Kai

0 Kudos
JesusG_Intel
Moderator
4,036 Views

Hello chi__kai,


I apologize for the precipitous closure of this thread.


The amount of paging you are seeing is not inconceivable since your processor has only 128MB of EPC and your data is 600MB. It all depends on how you are accessing the data within your enclave. If you are doing searches on your data, you may incur a lot of paging by accessing the 600MB often.


Sincerely,

Jesus G.

Intel Customer Support



0 Kudos
kai__chi
Novice
4,018 Views

Hi Jesus,

Yes, I totally agree that it depends on the data access patterns. I would just like to confirm that the numbers are not off. Vinutha recommended to measure MEM_INST_RETIRED.STLB_MISS_LOADS and  MEM_INST_RETIRED.STLB_MISS_STORES, which I did but I don't know how to interpret the results. I would be grateful if Vinutha or you could help me with that.

 

Best,

Kai

0 Kudos
JesusG_Intel
Moderator
3,991 Views

Hello kai__chi,


We are still investigating this issue. We will respond to this thread as soon as we have an update.


Sincerely,

Jesus G.

Intel Customer Support


0 Kudos
JesusG_Intel
Moderator
3,927 Views

Hello chi__kai,


We are still investigating this issue. We apologize for the delay.


Sincerely,

Jesus G.

Intel Customer Support


0 Kudos
JesusG_Intel
Moderator
3,780 Views

Hello Kai_chi,

 

I apologize for the long delay. Unfortunately, we have been unable to determine a method for counting EPC paging/hits/misses using VTune.

 

It seems that your estimate of 6GB is reasonable if you are traversing your array often.

 

Sincerely,

Jesus G.

Intel Customer Support

 

0 Kudos
JesusG_Intel
Moderator
3,747 Views

Hello kai_chi,


Since we do not have a solution to your question, we will close this thread. I apologize for the long wait only to be told at the end that we do not have an answer. If you have other SGX-related questions, please open a new thread and we will be happy to help you.


Sincerely,

Jesus G.

Intel Customer Support


0 Kudos
kai__chi
Novice
3,735 Views

Hi Jesus,

Thanks a lot for your time and responses.

Thanks for confirming that VTune is not capable of doing it. I will try to get more information about the two TLB hardware events and see if I can use this data effectively.

 

Best,

Kai

0 Kudos
Reply