Is it possible that UPI bandwidth “miscounted” into memory bandwidth due to its architecture design?

[SylarTian]

Hi,

I found that when I put the network threads on the same socket with NIC, there will be low memory bandwidth observed (just the same as when the system is idle); But if I put the network threads on the other socket, the bandwidth will increase just the similar figure as UPI incoming data traffic observed by PCM. I wonder if the PCM count the UPI traffic into memory bandwidth because of its architecture design. 

— I am currently working on a 2S-icelake system with 24 cores and 12 channels for each socket. I tried to run a network benchmark on either of the sockets and observe the system’s overall memory bandwidth with PCM. 

— The expected memory bandwidth of the network benchmark should be zero, since it does not process the data received and DDIO is fine-tuned to eliminate the effect of NIC’s DMA.

— And BTW, there are too few documents on illustrating UPI compared to QPI, I would appreciate it if any extra materials provided that can help me understand how UPI works.

Best regards,

Sylar

[AndrewG_Intel]

Hello @SylarTian

Thank you for posting on the Intel® communities.

In order to make sure we have the proper information and your inquiries are addressed by the proper channel of support, could please confirm the following?

1. When you said "2S-icelake system" do you mean you are using Intel® Xeon® Scalable Processors? Please confirm the model/SKU of the Processors on the system:
2. Also, please provide more details about the other hardware on the system. For instance, Server/PC/motherboard maker and model; Network Cards (NICs) maker, model, driver version; Operating System version, etc.:
3. When you said "UPI" and "QPI", are you referring to Intel® Quick Path Interconnect (Intel® QPI) and Intel® Ultra Path Interconnect (Intel® UPI)?

Best regards,

Andrew G.

Intel Customer Support Technician

[SylarTian]

Hi @AndrewG_Intel 

Thank you for your quick reply!

1. Processor model on my system: Intel(R) Xeon(R) Platinum 8331C CPU @ 2.50GHz.

2. Motherboard model on my system:  Manufacturer: Inspur Group; Product Name: SA5226H6; Ethernet controller: Mellanox Technologies Device 101d; os version: centOS-7.8; kernel version: 4.19

3. Yes, I am referring to intel ultra path interconnect. I think when I put threads on the different socket against NIC's, the higher UPI traffic results in higher memory bandwidth. But this part of memory bandwidth is not valid, since UPI should not access DRAM with all the DMA data kept in LLC by intel DDIO. So my guess is cross-socket UPI traffic pass through memory controller on the dest socket, and some part of it is taken into memory bandwidth's account by intel pcm/Vtune. I wonder if you can confirm my speculation or illustrate why I am wrong.

Thanks!

Best regards,

Sylar

[AndrewG_Intel]

Hello SylarTian

Thank you for your response and for the details. Please allow me to route this to the proper channel of support to assist you with these concerns.

Best regards,

Andrew G.

Intel Customer Support Technician

[Victor_G_Intel]

Hello SylarTian,

Thank you for posting on the Intel® communities.

To continue with your request can you please provide the following information:

1. Are you doing these tests on behalf of a company? If yes, please provide as many details as possible about the company that you are doing these tests for.
2. Can you please let us know which tool you were using to observe the system’s overall memory bandwidth during the benchmark tests? You mentioned PCM; however, we would like to know if it was either the Intel® Performance Counter Monitor or the Processor Counter Monitor  
3. Would you be able to provide us with a copy of the software that you used to complete the benchmarking tests? If yes, please provide us as well the exact steps that you or your team took during the tests and all the environmental details involved in the tests in order to analyze everything on our end.
4. Is the Intel® Xeon Platinum 8331C the only processor that you have used for these tests?
5. Is there any documentation that you started investigating before going into the actual tests?

Regards,

Victor G.

Intel Technical Support Technician  

[SylarTian]

Hi @Victor_G_Intel

Thank you for your reply.

1. The main goal of this test is to reduce memory bandwidth as much as possible. While during the test, we find that there will be unexpected increment of memory bandwidth when deploying the threads on the different socket against NIC and UPI has the same trend.

2. I am using both intel Vtune and Processor Counter Monitor to observe the performance of my benchmark which gave me similar results on those tests.

4. Yes.

5. Most documentations I referred during my investigation are from intel website (white paper/community discussion/cookbook), and they are mainly about intel DDIO technology & cache coherence protocol & UPI & NUMA node. (NUMA is off during my tests)

 I am sorry that I cannot provide the other information. Hope these information could help.

Best regards,

Sylar

[Victor_G_Intel]

Hello SylarTian,

Thank you for your response.

Please let me review this information internally, and kindly wait for an update.

Once we have more information to share, we will post it on this thread.

Regards,

Victor G.

Intel Technical Support Technician  

[Victor_G_Intel]

Hello SylarTian,

Thank you so much for your patience.

To continue with our investigation, we have requested some extra pieces of information via private message. Please respond through the private message and not through the forum so we can continue our research.

Best regards,

Victor G.

Intel Technical Support Technician  

[Paul_R_Intel]

Hello SylarTian,

I hope you are doing great, we would like to know if you received the private message as we have not seen an update from your end.

We will be waiting for your response.

Paul R.

Intel Technical Support Technician  

[BrusC_Intel]

Hello, SylarTian.

Good day,

I wanted to follow up on your thread to confirm if you had time to share the information we previously requested.

We will keep the thread open and wait for your response.

I will follow up on March 31st just in case.

Best regards,

Bruce C.

Intel Customer Support Technician

[Victor_G_Intel]

Hello SylarTian,

We have not heard back from you.

If you need any additional information, please submit a new question as this thread will no longer be monitored 

Regards,

Victor G. 

Intel Technical Support Technician