X710-DA2 10GbE SFP+ never reaches full uplink throughput

[JacobW]

Hi,

I have a problem with a X710-DA2 NIC that doesn't reach full uplink throughput.

iperf3 -c <iperf server> -b 10G
Connecting to host <iperf server> port 5201
[ 5] local <iperf client> port 45200 connected to <iperf server> port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.08 GBytes 9.32 Gbits/sec 73 1.17 MBytes
[ 5] 1.00-2.00 sec 1.09 GBytes 9.39 Gbits/sec 73 502 KBytes
[ 5] 2.00-3.00 sec 1.09 GBytes 9.39 Gbits/sec 82 561 KBytes
[ 5] 3.00-4.00 sec 1.09 GBytes 9.39 Gbits/sec 43 577 KBytes
[ 5] 4.00-5.00 sec 1.09 GBytes 9.40 Gbits/sec 84 378 KBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.39 Gbits/sec 59 1.01 MBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.39 Gbits/sec 59 1.24 MBytes
[ 5] 7.00-8.00 sec 1.09 GBytes 9.39 Gbits/sec 57 1.28 MBytes
[ 5] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 83 1.26 MBytes
[ 5] 9.00-10.00 sec 1.09 GBytes 9.39 Gbits/sec 66 543 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec 679 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.38 Gbits/sec receiver

iperf3 -R -c <iperf server> -b 10G
Connecting to host <iperf server>, port 5201
Reverse mode, remote host <iperf server> is sending
[ 5] local <iperf client>port 45204 connected to <iperf server> port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 589 MBytes 4.94 Gbits/sec
[ 5] 1.00-2.00 sec 636 MBytes 5.33 Gbits/sec
[ 5] 2.00-3.00 sec 658 MBytes 5.52 Gbits/sec
[ 5] 3.00-4.00 sec 633 MBytes 5.31 Gbits/sec
[ 5] 4.00-5.00 sec 636 MBytes 5.34 Gbits/sec
[ 5] 5.00-6.00 sec 597 MBytes 5.01 Gbits/sec
[ 5] 6.00-7.00 sec 660 MBytes 5.54 Gbits/sec
[ 5] 7.00-8.00 sec 704 MBytes 5.91 Gbits/sec
[ 5] 8.00-9.00 sec 592 MBytes 4.97 Gbits/sec
[ 5] 9.00-10.00 sec 644 MBytes 5.40 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 6.20 GBytes 5.33 Gbits/sec 1331 sender
[ 5] 0.00-10.00 sec 6.20 GBytes 5.33 Gbits/sec receiver

03:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710-2
Kernel driver in use: i40e
Kernel modules: i40e
03:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
Subsystem: Intel Corporation Ethernet Converged Network Adapter X710
Kernel driver in use: i40e
Kernel modules: i40e

Settings for ens5f0:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseX/Full
10000baseLR/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseX/Full
10000baseLR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

Comparing to another machine using a X520 where we don't have issues with reaching full throughput.

04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Kernel driver in use: ixgbe
Kernel modules: ixgbe
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Kernel driver in use: ixgbe
Kernel modules: ixgbe

Settings for ens5f1:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

There are differences in the configuration such as  'Advertised pause frame use' and 'Auto-negotiation'. I have tried to change these with ethtool but I ahve been unable to change the settings so far.

Any idea if the disprepancies in the settings is the cause and how to resolve this problem?

Thanks

/JW

[AlfredoS_Intel]

Hi Jacobw,

Thank you for posting in our Intel® Ethernet Communities Page.

We are sorry to hear about the issue that you are experiencing with your network adapter.

To better assist you, we need to get some logs from your system. It will tell us the different driver version and components installed on your system. Please download and run our Intel® System Support Utility from this page, https://downloadcenter.intel.com/download/26735/Intel-System-Support-Utility-for-the-Linux-Operating-System#:~:text=Intel%20SSU%20for%20the%20Linux,and%20shared%20by%20the%20user. After running it, you will be given an option to save the logs to a text file, please do so and attach the file on your reply.

Additionally, please provide us the results of this command: ethtool -i ethx, where ethx is the Ethernet port.

We look forward to hearing from you. If we do not get your reply, we will follow up after 3 business days.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Jacobw,

We are just following up.

It looks like you need more time to provide the information that we asked.

We will follow up again after 3 business days. Should we not hear from you, our system may automatically close the thread.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Jacobw, 

We need to close this thread since we have not gotten a response from you: maybe because you are busy or preoccupied at the moment. We know that this is important for you to get it resolved and it is also equally important for us to give you the right solution; as much as we would like to assist you, we need to close it to attend to other customers. We hope for your consideration and understanding on this one.

If you need any additional information, please submit a new question as this thread will no longer being monitored.

Thank you for contacting Intel® and have a great week!

Best Regards,

Alfred S

Intel® Customer Support