Processors
Intel® Processors, Tools, and Utilities
14663 Discussions

Xeon CPU splits PCIe MRd into mutiple smaller MRd

JCIe
Beginner
1,501 Views

Hi,

Im Working on an system where data is transferred between 2 PCIe units. The system consists of a requester and a completer. The requester will issue MRd packets to the completer. For optimizations purposes I want to the MRd packets arriving at the completer to be as big as possible. My problem is that each outgoing MRd from the requester arrives as multiple smaller MRds at the completer. PCIe analyzer was used to verify the results. All units are connected to PCIe directly under the CPU.

Is there any setting I can tweak to get rid of this behavior?


System info
CPU: 2x Xeon 4210r
Also tried a system with a single Xeon E5 1620 V3 with the same results.

Requester
Max payload: 256
Max readreq: 512
size of outgoing MRd: 256


Completer
Max payload: 256
Max readreq: 512
size of incoming MRd: 64

0 Kudos
10 Replies
SergioS_Intel
Moderator
1,456 Views

Hello JCIe,


Thank you for contacting Intel Customer Support.

 

Please allow us to check on your question and we will get back to you as soon as possible.



Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


0 Kudos
SergioS_Intel
Moderator
1,437 Views

Hello JCIe,


In order to continue assisting you please help us providing more additional information:


1. Was this working before?

2. What software are you using?

3. Can you please provide us the model of the server? 

4. Did you get assistance from the manufacturer?

5. Can you please provide us a screenshot of the error and logs information?


We will be looking forward to your reply.


Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


0 Kudos
JCIe
Beginner
1,415 Views

1. It has always been working like this on the mentioned intel systems. To be clear this is not an error but a behavior that hurts performance of the hardware we are constructing.

2. I dont't think the software matters sincen this occurs PCIe TLP level. I've been using SPDK with an NVME drive as requester and Cuda with an Nvidia GPU as requester.

3. The systems is based on a supermicro X11DPi-NT motherboard.

4. No.

5. There are no logs but I can post som screenshots of the TLPs from the PCIe analyzer the next time I do measurements.

0 Kudos
JCIe
Beginner
1,400 Views

Attached is analyzer info

MRd_requster: Outgoing MRd captured at disk (requester) slot.

MRd_completer: Incoming MRd captured at completer slot.


Both units are connected to PCIe slots going directly to (the same) CPU. As you can see the root complex act as a middle man and splits each MRd into multiple smaller MRds. I would like the MRds to keep their original size when being forwarded.

0 Kudos
SergioS_Intel
Moderator
1,425 Views

Hello JCie,


We are following your thread and we would like to know if you were able to gather the information that we requested on our previous thread.


Best regards,


Sergio S.


Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios



0 Kudos
SergioS_Intel
Moderator
1,397 Views

Hello JCIe,


We appreciated the additional information, please allow us to check it and we will get back to you.


Best regards,


Sergio S.


Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios



0 Kudos
SergioS_Intel
Moderator
1,380 Views

Hello JCIe,


We would like to inform you that we are still working on your inquiry and will send you an update earlier next week.


Additionally, can you please let us know why you need this information? Are you evaluating for purchase of processors/systems, etc.?

 

We will be looking forward to your updates.



Best regards,

Sergio S.

Intel Customer Support Technician


For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


0 Kudos
JCIe
Beginner
1,373 Views

Great.

Yes, that is somewhat the case, we construct systems involving high performance data transfer. This behavior will limit Intel CPUs to only be used for lower performing systems.

0 Kudos
SergioS_Intel
Moderator
1,348 Views

Hello JCIe,


We appreciate the additional information, we are actively working on your case and will send you an update as soon as possible.

 

Best regards,

Sergio S.

Intel Customer Support Technician


For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


0 Kudos
SergioS_Intel
Moderator
1,218 Views

Hello JCIe,


We are going to send you a private message in order to continue working on your question.



Best regards,

Sergio S.

Intel Customer Support Technician


For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


0 Kudos
Reply