Why these BIOS options will impact RDMA performance

[AARONMJ_L_]

Hi All, I am testing the performance of RDMA write to NVDIMM via NTB on our storage server. There are two nodes connected through PLX switch's NTB and each node has two Skylake sockets and four RDIMMs and one NVDIMM. And the DMA engine we used is on PLX switch which connect to socket 1. I've tried to test two different scenarios with default BIOS setting: 1) RDMA write to NVDIMM on socket 0 --> NTB link bandwidth is about 6.5 GB/s and link utilization is about 43%. 2) RDMA write to NVDIMM on socket 1 --> NTB link bandwidth is about 13.3 GB/s and link utilization is about 89%. But when I change the "PCIe allocating write" option in BIOS to "Non Allocating Write", the test result as below: 1) RDMA write to NVDIMM on socket 0 --> NTB link bandwidth is about 12.2 GB/s and link utilization is about 81.8%. 2) RDMA write to NVDIMM on socket 1 --> NTB link bandwidth is about 12.18 GB/s and link utilization is about 81.7%. And I also try to change "IO Directory Cache" option in BIOS to "Enable for Remote InvItoM Hybrid AllocNonAlloc" and the test result as below: 1) RDMA write to NVDIMM on socket 0 --> NTB link bandwidth is about 12.11 GB/s and link utilization is about 81.25%. 2) RDMA write to NVDIMM on socket 1 --> NTB link bandwidth is about 13.35 GB/s and link utilization is about 89.5%. As you can see, if only use default BIOS setting, the RDMA write to NVDIMM on socket 0 will get bad performance. If "PCIe allocating write" option use "Non Allocating Write", the performance on two scenarios are almost the same. If "IO Directory Cache" option use "Enable for Remote InvItoM Hybrid AllocNonAlloc", the RDMA write to NVDIMM on socket 0 will improve from 6.5 GB/s to 12.11 GB/s. Can someone please explain the difference between those BIOS options and why cause this result? PCIe allocating write IO Directory Cache Or point me to relevant documentation. Thanks you, Paul

[Karl_K_2]

We observed the similar issue, and are pending on Intel's further response. One reference pointer is Intel's DDIO, though document about the DDIO was compiled back to 2012. We are not sure the outdated DDIO document is still fully validated to current Purley platform.