Intel® ISA Extensions
Use hardware-based isolation and memory encryption to provide more code protection in your solutions.

DMA throughput evaluation

a1rjun
Beginner
857 Views

Hello,

We are developers currently testing our 100G NIC (Mellanox connect X5) on our Intel(R) Xeon(R) Gold 6258R CPU @ 2.70GHz (4 NUMA)running Ubuntu18.04 and have some questions regarding the DMA to be used. We have 2 options:

1)  Use DMA available on the NIC (NIC connected via PCIe bus ).

2) Use the host DMA engine

Host DMA specs are as follows: 

System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07)
Subsystem: Intel Corporation Sky Lake-E CBDMA Registers
Kernel driver in use: ioatdma
Kernel modules: ioatdma

There are 16 such registers(8 per NUMA) and while checking the DMA channels, we can see 16 :

$ ls -1 /sys/class/dma/
dma0chan0
dma10chan0
dma11chan0
dma12chan0
dma13chan0
dma14chan0
dma15chan0
dma1chan0
dma2chan0
dma3chan0
dma4chan0
dma5chan0
dma6chan0
dma7chan0
dma8chan0
dma9chan0

While checking the throughput using the kernel module dmatest.ko , we got the following results:

$sudo insmod ./dmatest.ko channel=dma0chan0 timeout=2000 iterations=100 run=1
$
$ dmesg
[783764.045909] dmatest: Started 1 threads using dma0chan0
[783764.045973] dmatest: dma0chan0-copy: summary 100 tests, 0 failures 20308 iops 172420 KB/s (0)
 
The throughput achieved is around 172420 KBps (1.4Gbps) for a single DMA channel .We also tried using taskset to run on a single core.
$ taskset -c 4 sudo insmod ./dmatest.ko channel=dma0chan0 timeout=2000 iterations=100 run=1
but got similar results.
 
As far as our understanding, the NIC is connected to the memory controller through one DMA(Host) channel. So for evaluating the throughput of one DMA channel, we used the above mentioned method.
 

Is our understanding correct? If not, can we use multiple host DMA channels for a single device(NIC) ?  
Since the throughput from one DMA channel is very low(1.4Gbps) compared to our expectations (100Gbps), we raised this query.

Additional queries:
1) What is the DMA controller(chip) used in this setup? is it 8237?
2) Is this the throughput to be expected from a single DMA channel?
3) If not, then what's the limit and how to achieve it? 
 
 
Regards,
Arjun
0 Kudos
0 Replies
Reply