Emerald lake Xeon scalable memory directory S state V.S home snoop ( memory directory disabled)

[Jerryidk]

Hi, all 

I have observed strange performance drop on read workload when the cachelines accessed from with memory directory state S on emerald lake product. From my understanding, when memory directory mode is enabled and cacheline with memory directory state is S (shared), a load miss will sent out a snoop request on UPI link, which should be very similar to always snoop (disable memory directory) mode. However, I have observed that under heavy memory read workload. The performance with S state marked workload drop significantly. There seems to be a hardware problem, I am not sure what. 

Machine configuration/stats :

• processor model : XEON GOLD 6548Y+
• 2 socket set up
• max upi bw per direction 120 GB/s
• max mem bw: 250 GB/s (single socket)

I have attached a screenshot collected by vtune. My experiment set up is relatively simple. I set up a array of cachelines (about 4 GB), and then spawn all 32 cpus to pseudorandomly access cachelines base on numa nodes I wanted. For example, when I say local read, meaning all threads allocated on local numa node are pseudorandomly reading the memory array. Similar for remote read, except all threads are allocated from the remote socket. I won't explain the details of how I was able to change cacheline memory directory bit to S, but I am confident S dir bit is set because I was able to observe the corresponding PMU counter for S state. 

This is my first time posting, so please let me know if you would like more info. 

Thank you @McCalpinJohn for all the great information you have provided on this subject, I couldn't even get to this step without the information you have provided. 

[Azhari_Intel]

Hi Jerryidk,

Thank you for reaching Intel Community.

Please refer to the Intel® 64 and IA-32 Architectures Software Developer’s Manual for background information that may help in understanding memory directory behavior:

https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html

Since the processor in question is a tray processor, we recommend reaching out to your place of purchase or engaging with your Intel representative for further technical assistance.

Best Regards,

Azhari_Intel

[Azhari_Intel]

Hi Jerryidk,

Hope you are well.

Just wanted to follow up with you.

Kindly let us know if you have any further concerns.

Looking forward to your response.

Regards,

Azhari_Intel