Need help with Xeon CPU Interconnect / UPI monitoring

[SimonHF]

Hello!

I have a Sapphire Rapids system with dual "Intel(R) Xeon(R) Platinum 8480+" processors running Linux. Unfortunately I'm running a huge variety of nearly all multi threaded software (both off-the-shelf and home grown) which is not NUMA aware. In this case the kernel can schedule threads accessing the same memory on different sockets, which means threads may have to access remote memory via the interconnect between the sockets, which also worsens performance. There are ways to help the software run better, e.g. pinning the software to the cores of a particular socket so that in theory remote memory is never used or accessed. Are there other additional ways too?

As an example, running benchmarks on postgreSQL pinned and unpinned results in up to a 11.1% performance difference. Presumably mostly due to remote memory access via the interconnect?

Presumably the performance overhead of the remote memory access via the interconnect will depend on the particular software application itself and its memory usage patterns?

On a particular host, I would like to be able to monitor interconnect usage with the idea that more usage means software which hasn't been pinned correctly / "saved" from using remote memory. Is this idea realistic? I'm assuming it is.

In order to try and achieve this, I've been on the hunt for software which can report the amount of interconnect usage and I found the Intel Performance Counter Monitor (AKA Intel PCM) tool [1]. Are there any other alternative tools?

To test Intel pcm, I created a simple C program to cause memory to be synchronized across the interconnect, which in tests runs up to 37% slower with interconnect usage. However, when running Intel pcm and the simple C program exercising the interconnect, at the same time, the interconnect monitoring / reporting from Intel pcm wasn't obviously showing me there was a problem / the extra interconnect usage. I also looked at some other Intel pcm metrics relating to memory, but they also weren't obviously showing me there was a problem, either. FYI I posted the steps to show Intel pcm not being useful for what I wanted here [2], including the simple C program. But no response there so far.

So currently I have  found no way to effectively monitor interconnect usage when I am running a program specifically designed to generate interconnect usage.

I am posting here in the hope that interconnect experts will chime in and help, or put me right if I am misunderstanding how these things work or whether it's possible to detect what I want to detect at all.

Thanks in advance!

[1] https://github.com/intel/pcm

[2] https://github.com/intel/pcm/issues/919

[NormanS_Intel]

Hello SimonHF,

Thank you for posting in the community!

To ensure you receive the most specialized assistance, we have a dedicated forum that addresses these specific concerns. Therefore, I will be moving this discussion to our Server Forum. This will allow our knowledgeable community and experts to provide you with timely and accurate solutions.

Best regards,

Norman S.

Intel Customer Support Engineer

[Ragulan_Intel]

Hello SimonHF,

Thank you for reaching out to Intel Customer Support!

As pertaining to UPI, the links below would hopefully provide a better insight on your concern.

1. https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/2nd-gen-xeon-scalable-datasheet-vol-1.pdf
2. https://edc.intel.com/content/www/us/en/design/products-and-solutions/processors-and-chipsets/eagle-stream/4th-gen-xeon-processor-scalable-family-code-name-sapphire-rapids-data/
3. Where to Find the Intel® Ultra Path Interconnect (Intel® UPI) Speed...
4. What Is the Difference in Cache Memory Between CPUs for Intel® Xeon®...
5. https://cdrdv2.intel.com/v1/dl/getContent/606154?explicitVersion=true
6. https://cdrdv2.intel.com/v1/dl/getContent/560113?explicitVersion=true

Note: Some of the above documents would require you to have privileged access that you can apply for using the steps in the following link: https://www.intel.com/content/www/us/en/support/articles/000058073/programs/resource-and-documentation-center.html

Thank You & Best Regards,

Ragulan_Intel

[SimonHF]

Hello Ragulan_Intel,

Thanks for the response.
I had a look at the links  you kindly provided:

Link 1: Appears to be a data sheet.
Link 2: Appears to be a data sheet.
Link 3: Is a redirect to Intel ARK.
Link 4: Talks about Xeon cache memory differences.
Links 5 & 6: Do not work for me. They take me to the "Intel Azure Portal". If I enter my "Intel Developer Zone Standard" account -- which I created using the above link given in order for privileged access -- then it says "Your account or password is incorrect". And if I enter my Microsoft account it says "AADSTS90123: The token can't be issued because the identity or claim issuance provider denied the request. Response code: access_denied.".

So it seems like the links 1 to 4 offer no advice on tools or strategies to detect and or minimize interconnect traffic. And links 5 & 6 don't work for me

In case the above post was generated by AI, it would be great to hear from a human!

Thanks,
Simon

[MACM]

HI SimonHF,

Greetings of the day.

Hope you are doing great.

Thank you for your response.

Please allow us some time to check this internally and get back to you with an update.

Best Regards,

Mohammed Ali CM

Intel Customer Support

[pujeeth]

Hi SimonHF,

Greetings of the day.

We see that our Intel Engineering support has responded on GitHub: https://github.com/intel/pcm/issues/919.

This will be the proper path for your question and the best option for development assistance. We recommend that you continue the conversation through that portal for better assistance.

Since there are no pending actions for this thread, we will proceed to archive it.

Thank you for your understanding.

Best regards,

Pujeeth_Intel.