Emerald lake Xeon scalable memory directory S state V.S home snoop ( memory directory disabled)

[Jerryidk]

Hi, all 

I have observed strange performance drop on read workload when the cachelines accessed from with memory directory state S on emerald lake product. From my understanding, when memory directory mode is enabled and cacheline with memory directory state is S (shared), a load miss will sent out a snoop request on UPI link, which should be very similar to always snoop (disable memory directory) mode. However, I have observed that under heavy memory read workload. The performance with S state marked workload drop significantly. There seems to be a hardware problem, I am not sure what. 

Machine configuration/stats :

• processor model : XEON GOLD 6548Y+
• 2 socket set up
• max upi bw per direction 120 GB/s
• max mem bw: 250 GB/s (single socket)

I have attached a screenshot collected by vtune. My experiment set up is relatively simple. I set up a array of cachelines (about 4 GB), and then spawn all 32 cpus to pseudorandomly access cachelines base on numa nodes I wanted. For example, when I say local read, meaning all threads allocated on local numa node are pseudorandomly reading the memory array. Similar for remote read, except all threads are allocated from the remote socket. I won't explain the details of how I was able to change cacheline memory directory bit to S, but I am confident S dir bit is set because I was able to observe the corresponding PMU counter for S state. 

This is my first time posting, so please let me know if you would like more info. 

Thank you @McCalpinJohn for all the great information you have provided on this subject, I couldn't even get to this step without the information you have provided. 

[Azhari_Intel]

Hi Jerryidk,

Thank you for reaching Intel Community.

Please refer to the Intel® 64 and IA-32 Architectures Software Developer’s Manual for background information that may help in understanding memory directory behavior:

https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

Since the processor in question is a tray processor, we recommend reaching out to your place of purchase or engaging with your Intel representative for further technical assistance.

Best Regards,

Azhari_Intel

[Azhari_Intel]

Hi Jerryidk,

Hope you are well.

Just wanted to follow up with you.

Kindly let us know if you have any further concerns.

Looking forward to your response.

Regards,

Azhari_Intel

[Jerryidk]

Hi, Azhari

Could you maybe further identify the section in developer manual that talk about memory directory. 

Also, maybe let me rephrase my question here.  Why is in memory directory mode, accessing a cacheline with S state

cost more UPI bandwidth than always snoop ? What is the extra traffic here ?

Best,

Jerry

[Amina_Sadiya]

Hi Jerryidk,

Thank you for your response, Since it is a tray processor, we recommend reaching out to your place of purchase or engaging with your Intel representative for further technical assistance. 

Best regards,

Amina

Intel Customer Support Technician

[Azhari_Intel]

Hi Jerryidk,

Good day to you.

Just wanted to follow up with you on the reported issue. Please let us know if you have any further concerns.

Regards,

Azhari_Intel

[Azhari_Intel]

Hi Jerryidk,

Greetings and good day to you!

Since we have not seen an update, we will be closing this forum case from our end.

To sum it up, you may refer to our public documentation available that may explain the issue you are facing.

Architectures Software Developer Manuals: https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

Product Brief: https://www.intel.com/content/www/us/en/products/sku/237564/intel-xeon-gold-6548y-processor-60m-cache-2-50-ghz/specifications.html

As it is a tray processor, we recommend reaching out to your place of purchase or engaging with your Intel representative for further technical assistance.

If you need any additional information or assistance, please submit a new question as this thread will no longer be monitored.

Regards,

Azhari_Intel