Xeon CPU splits PCIe MRd into mutiple smaller MRd

[JCIe]

Hi,

Im Working on an system where data is transferred between 2 PCIe units. The system consists of a requester and a completer. The requester will issue MRd packets to the completer. For optimizations purposes I want to the MRd packets arriving at the completer to be as big as possible. My problem is that each outgoing MRd from the requester arrives as multiple smaller MRds at the completer. PCIe analyzer was used to verify the results. All units are connected to PCIe directly under the CPU.

Is there any setting I can tweak to get rid of this behavior?

System info
CPU: 2x Xeon 4210r
Also tried a system with a single Xeon E5 1620 V3 with the same results.

Requester
Max payload: 256
Max readreq: 512
size of outgoing MRd: 256

Completer
Max payload: 256
Max readreq: 512
size of incoming MRd: 64

[SergioS_Intel]

Hello JCIe,

Thank you for contacting Intel Customer Support.

Please allow us to check on your question and we will get back to you as soon as possible.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[SergioS_Intel]

Hello JCIe,

In order to continue assisting you please help us providing more additional information:

1. Was this working before?

2. What software are you using?

3. Can you please provide us the model of the server? 

4. Did you get assistance from the manufacturer?

5. Can you please provide us a screenshot of the error and logs information?

We will be looking forward to your reply.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[JCIe]

1. It has always been working like this on the mentioned intel systems. To be clear this is not an error but a behavior that hurts performance of the hardware we are constructing.

2. I dont't think the software matters sincen this occurs PCIe TLP level. I've been using SPDK with an NVME drive as requester and Cuda with an Nvidia GPU as requester.

3. The systems is based on a supermicro X11DPi-NT motherboard.

4. No.

5. There are no logs but I can post som screenshots of the TLPs from the PCIe analyzer the next time I do measurements.

[JCIe]

Attached is analyzer info

MRd_requster: Outgoing MRd captured at disk (requester) slot.

MRd_completer: Incoming MRd captured at completer slot.

Both units are connected to PCIe slots going directly to (the same) CPU. As you can see the root complex act as a middle man and splits each MRd into multiple smaller MRds. I would like the MRds to keep their original size when being forwarded.

[SergioS_Intel]

Hello JCie,

We are following your thread and we would like to know if you were able to gather the information that we requested on our previous thread.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[SergioS_Intel]

Hello JCIe,

We appreciated the additional information, please allow us to check it and we will get back to you.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[SergioS_Intel]

Hello JCIe,

We would like to inform you that we are still working on your inquiry and will send you an update earlier next week.

Additionally, can you please let us know why you need this information? Are you evaluating for purchase of processors/systems, etc.?

We will be looking forward to your updates.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[JCIe]

Great.

Yes, that is somewhat the case, we construct systems involving high performance data transfer. This behavior will limit Intel CPUs to only be used for lower performing systems.

[SergioS_Intel]

Hello JCIe,

We appreciate the additional information, we are actively working on your case and will send you an update as soon as possible.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[SergioS_Intel]

Hello JCIe,

We are going to send you a private message in order to continue working on your question.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios