Community
cancel
Showing results for 
Search instead for 
Did you mean: 
JCIe
Beginner
152 Views

Xeon CPU splits PCIe MRd into mutiple smaller MRd

Hi,

Im Working on an system where data is transferred between 2 PCIe units. The system consists of a requester and a completer. The requester will issue MRd packets to the completer. For optimizations purposes I want to the MRd packets arriving at the completer to be as big as possible. My problem is that each outgoing MRd from the requester arrives as multiple smaller MRds at the completer. PCIe analyzer was used to verify the results. All units are connected to PCIe directly under the CPU.

Is there any setting I can tweak to get rid of this behavior?


System info
CPU: 2x Xeon 4210r
Also tried a system with a single Xeon E5 1620 V3 with the same results.

Requester
Max payload: 256
Max readreq: 512
size of outgoing MRd: 256


Completer
Max payload: 256
Max readreq: 512
size of incoming MRd: 64

0 Kudos
8 Replies
SergioS_Intel
Moderator
107 Views

Hello JCIe,


Thank you for contacting Intel Customer Support.

 

Please allow us to check on your question and we will get back to you as soon as possible.



Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


SergioS_Intel
Moderator
88 Views

Hello JCIe,


In order to continue assisting you please help us providing more additional information:


1. Was this working before?

2. What software are you using?

3. Can you please provide us the model of the server? 

4. Did you get assistance from the manufacturer?

5. Can you please provide us a screenshot of the error and logs information?


We will be looking forward to your reply.


Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


SergioS_Intel
Moderator
76 Views

Hello JCie,


We are following your thread and we would like to know if you were able to gather the information that we requested on our previous thread.


Best regards,


Sergio S.


Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios



JCIe
Beginner
66 Views

1. It has always been working like this on the mentioned intel systems. To be clear this is not an error but a behavior that hurts performance of the hardware we are constructing.

2. I dont't think the software matters sincen this occurs PCIe TLP level. I've been using SPDK with an NVME drive as requester and Cuda with an Nvidia GPU as requester.

3. The systems is based on a supermicro X11DPi-NT motherboard.

4. No.

5. There are no logs but I can post som screenshots of the TLPs from the PCIe analyzer the next time I do measurements.

JCIe
Beginner
52 Views

Attached is analyzer info

MRd_requster: Outgoing MRd captured at disk (requester) slot.

MRd_completer: Incoming MRd captured at completer slot.


Both units are connected to PCIe slots going directly to (the same) CPU. As you can see the root complex act as a middle man and splits each MRd into multiple smaller MRds. I would like the MRds to keep their original size when being forwarded.

SergioS_Intel
Moderator
48 Views

Hello JCIe,


We appreciated the additional information, please allow us to check it and we will get back to you.


Best regards,


Sergio S.


Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios



SergioS_Intel
Moderator
31 Views

Hello JCIe,


We would like to inform you that we are still working on your inquiry and will send you an update earlier next week.


Additionally, can you please let us know why you need this information? Are you evaluating for purchase of processors/systems, etc.?

 

We will be looking forward to your updates.



Best regards,

Sergio S.

Intel Customer Support Technician


For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios


JCIe
Beginner
24 Views

Great.

Yes, that is somewhat the case, we construct systems involving high performance data transfer. This behavior will limit Intel CPUs to only be used for lower performing systems.