Slow speed on X520-DA2 cards under Windows 10

[xxx59712]

Hi,

The two workstation are connected directly by a DAC Copper cable (I ruled out the cable issue

by using 3 differents brands and cable lenght).

XA520-DA2 Carsd and Driver 4.1.197.0 - 27/11/2019 on both machines

Windows 10 Pro on both workstations.

The problem is that Bandwith results  are below expectations and not symmetrical and that they are well below what I get on Ubuntu. Using two Ubuntu live USB keys and doing a Iperf on both machines, I get a constant robust symmetrical 9.49 Gbits/sec between the two workstations.

I ruled out the PCIe ports restrictions by leaving just the NIC on the 16x port of each machine and by using a Remote Desktop to connect to Windows.

Here are the results on Windows. I tried every possible settings on the Device manager Driver tab. The only thing that seems to improve things a bit are Disabling Flow Control and setting the Interrupt Moderation Rate to Off. The other settings, jumbo frame and other Buffers have zero impact on the performance.

>iperf3 -c 10.10.10.3
Connecting to host 10.10.10.3, port 5201
[ 4] local 10.10.10.2 port 9356 connected to 10.10.10.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 604 MBytes 5.07 Gbits/sec
[ 4] 1.00-2.00 sec 678 MBytes 5.69 Gbits/sec
[ 4] 2.00-3.00 sec 731 MBytes 6.13 Gbits/sec
[ 4] 3.00-4.00 sec 745 MBytes 6.25 Gbits/sec
[ 4] 4.00-5.00 sec 747 MBytes 6.27 Gbits/sec
[ 4] 5.00-6.00 sec 742 MBytes 6.22 Gbits/sec
[ 4] 6.00-7.00 sec 769 MBytes 6.45 Gbits/sec
[ 4] 7.00-8.00 sec 764 MBytes 6.41 Gbits/sec
[ 4] 8.00-9.00 sec 758 MBytes 6.36 Gbits/sec
[ 4] 9.00-10.00 sec 788 MBytes 6.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 7.16 GBytes 6.15 Gbits/sec sender
[ 4] 0.00-10.00 sec 7.16 GBytes 6.15 Gbits/sec receiver

iperf Done.

C:\Users\gdn\Desktop\iperf-3.1.3-win64>iperf3 -c 10.10.10.2
Connecting to host 10.10.10.2, port 5201
[ 4] local 10.10.10.3 port 49799 connected to 10.10.10.2 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 324 MBytes 2.72 Gbits/sec
[ 4] 1.00-2.00 sec 334 MBytes 2.80 Gbits/sec
[ 4] 2.00-3.00 sec 350 MBytes 2.93 Gbits/sec
[ 4] 3.00-4.00 sec 327 MBytes 2.74 Gbits/sec
[ 4] 4.00-5.00 sec 324 MBytes 2.72 Gbits/sec
[ 4] 5.00-6.00 sec 276 MBytes 2.31 Gbits/sec
[ 4] 6.00-7.00 sec 321 MBytes 2.69 Gbits/sec
[ 4] 7.00-8.00 sec 323 MBytes 2.71 Gbits/sec
[ 4] 8.00-9.00 sec 336 MBytes 2.82 Gbits/sec
[ 4] 9.00-10.00 sec 336 MBytes 2.82 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.17 GBytes 2.73 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.17 GBytes 2.73 Gbits/sec receiver

iperf Done.

Open to any suggestions to improve the situation.

Thank you.

[xxx59712]

Adding the same result done yesterday (same exact machine on both side) but with Ubuntu 20.04.

Connecting to host 10.10.10.3, port 5201
[ 5] local 10.10.10.2 port 59388 connected to 10.10.10.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.09 GBytes 9.32 Gbits/sec 85 1.77 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 7 1.77 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 10 1.77 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.77 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec 102 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver

iperf Done.

Connecting to host 10.10.10.2, port 5201
[ 5] local 10.10.10.3 port 46954 connected to 10.10.10.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.08 GBytes 9.32 Gbits/sec 37 1.36 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 2.00-3.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.36 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.37 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.37 MBytes
[ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.37 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.38 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 37 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver

iperf Done.

[xxx59712]

Oh my, I continue testing and with the same driver settings, different version of the Windows driver gives different results!!!!

The worst is the most recent one (25.1.1), the best (so far) is version 25.0 with 7.8/8 Gbits/sec.

The plot thickens.