- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi All,
I am testing the performance of RDMA write to NVDIMM via NTB on our storage server.
There are two nodes connected through PLX switch's NTB and each node has two Skylake sockets and four RDIMMs and one NVDIMM.
And the DMA engine we used is on PLX switch which connect to socket 1.
I've tried to test two different scenarios with default BIOS setting:
1) RDMA write to NVDIMM on socket 0 --> NTB link bandwidth is about 6.5 GB/s and link utilization is about 43%.
2) RDMA write to NVDIMM on socket 1 --> NTB link bandwidth is about 13.3 GB/s and link utilization is about 89%.
But when I change the "PCIe allocating write" option in BIOS to "Non Allocating Write", the test result as below:
1) RDMA write to NVDIMM on socket 0 --> NTB link bandwidth is about 12.2 GB/s and link utilization is about 81.8%.
2) RDMA write to NVDIMM on socket 1 --> NTB link bandwidth is about 12.18 GB/s and link utilization is about 81.7%.
And I also try to change "IO Directory Cache" option in BIOS to "Enable for Remote InvItoM Hybrid AllocNonAlloc" and the test result as below:
1) RDMA write to NVDIMM on socket 0 --> NTB link bandwidth is about 12.11 GB/s and link utilization is about 81.25%.
2) RDMA write to NVDIMM on socket 1 --> NTB link bandwidth is about 13.35 GB/s and link utilization is about 89.5%.
As you can see, if only use default BIOS setting, the RDMA write to NVDIMM on socket 0 will get bad performance.
If "PCIe allocating write" option use "Non Allocating Write", the performance on two scenarios are almost the same.
If "IO Directory Cache" option use "Enable for Remote InvItoM Hybrid AllocNonAlloc", the RDMA write to NVDIMM on socket 0 will improve from 6.5 GB/s to 12.11 GB/s.
Can someone please explain the difference between those BIOS options and why cause this result?
PCIe allocating write
IO Directory Cache
Or point me to relevant documentation.
Thanks you,
Paul
Link Copied
1 Reply
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
We observed the similar issue, and are pending on Intel's further response. One reference pointer is Intel's DDIO, though document about the DDIO was compiled back to 2012. We are not sure the outdated DDIO document is still fully validated to current Purley platform.

Reply
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page