DMA throughput evaluation

[a1rjun]

Hello we are using the Intel(R) Xeon(R) Gold 6258R CPU @ 2.70GHz (2 NUMA)running Ubuntu18.04 and the DMA specs are as follows: 

System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07)
Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
Kernel driver in use: ioatdma
Kernel modules: ioatdma

There are 16 such registers(8 per NUMA) and while checking the DMA channels, we can see 16 :

$ ls -1 /sys/class/dma/
dma0chan0
dma10chan0
dma11chan0
dma12chan0
dma13chan0
dma14chan0
dma15chan0
dma1chan0
dma2chan0
dma3chan0
dma4chan0
dma5chan0
dma6chan0
dma7chan0
dma8chan0
dma9chan0

While checking the throughput using the kernel module dmatest.ko , we are able to see the following:

$ sudo insmod ./dmatest.ko channel=dma0chan0 timeout=2000 iterations=100 run=1

$

$ dmesg

[783764.045909] dmatest: Started 1 threads using dma0chan0

[783764.045973] dmatest: dma0chan0-copy: summary 100 tests, 0 failures 20308 iops 172420 KB/s (0)

The throughput achieved is around 172420 KBps for a single DMA channel .We also tried using taskset to run on a single core.
$ taskset -c 4 sudo insmod ./dmatest.ko channel=dma0chan0 timeout=2000 iterations=100 run=1

but got similar results.

My queries:

1) What is the DMA controller(chip) used in this setup? is it 8237?

2) Is this the throughput to be expected from a single DMA channel?
3) If not, then what's the limit and how to achieve it? 

[SergioS_Intel]

Hello a1rjun,

Thank you for contacting Intel Customer Support.

Please allow us to check on your question, and we will get back to you as soon as possible.

Best regards,

Sergio S.

Intel Customer Support Technician

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[SergioS_Intel]

Hello a1rjun,

We appreciate your patience, in order to continue assisting you, can you please let us know the purpose of having this information?

Finally, can you please let us know if you are developers?

We will be looking forward to your reply.

Best regards,

Sergio S.

Intel Customer Support Technician

[a1rjun]

Hello Sergio,

We are developers currently testing our 100G NIC (Mellanox connect X5) on our Xeon server and we have 2 options for DMA

1) Use the host DMA engine

2) Use DMA available on the NIC (NIC connected via PCIe bus ).

As far as our understanding, the NIC is connected to the memory controller through one DMA(Host) channel. So for evaluating the throughput of one DMA channel, we used the above mentioned method.

Is our understanding correct? If not, can we use multiple host DMA channels for a single device(NIC) ?  
Since the throughput from one DMA channel is very low(172420 KBps ~1.4Gbps) compared to our expectations (100Gbps), we raised this query.

Regards,

Arjun 

[SergioS_Intel]

Hello a1rjun,

Thank you for the additional information, please allow us to check it, and we will get back to you as soon as possible.

Best regards,

Sergio S.

Intel Customer Support Technician

[SergioS_Intel]

Hello a1rjun,

In order to properly assist you with your request, please submit your questions to one of the following Intel Software Developer community forums:

https://community.intel.com/t5/Intel-ISA-Extensions/bd-p/isa-extensions

https://community.intel.com/t5/Intel-Moderncode-for-Parallel/bd-p/moderncode-parallel-architecture

Best regards,

Sergio S.

Intel Customer Support Technician

Note: Should you need further assistance from Intel regarding your inquiry, it would be appreciated if a response to this email is done, as opposed to sending a brand new email, unless it is for a different issue or inquiry. This will assure that we can continue working on the original issue.

For firmware updates and troubleshooting tips, visit :https://intel.com/support/serverbios

[a1rjun]

Hello Sergio,

Thankyou for your response. I have raised my questions in the new community link which you shared.

Regards,

Arjun

[SergioS_Intel]

Hello a1rjun,

You are more than welcome we are going to proceed and close this thread.

Best regards,

Sergio S.

Intel Customer Support Technician