X710-AT2 onboard not working on Ubuntu 22.04/Debian

[drv41]

Hello, I need help with the X710-AT2 (on an Asrock E3C256D4U-2L2T board). It doesn't work on Ubuntu 22.04/Debian/Proxmox, but it works under Feroda. So I think it is a problem with the operating system, drivers or network card configuration. I have the latest NVM (9.30), i40e (i40e-2.23.17) and iavf (iavf-4.9.1). LLDP along with all network management is disabled in BIOS. The network card is seen by the operating system, but I cannot get any traffic thought. I get rx_dropped packets on the interface, but rather none are transmitted. The interface tries to get an IP via DHCP, but mostly fails; sometimes it does, but no TCP transmission seems go thought. I managed to get it to work twice, probably by accident, when I experimented with the settings `ethtool -G eth12 rx 256 tx 256; ethtool -C ethX adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0`. The working state did not survive the reboot, and the commands that made it work (or anything else) did not make it work again.

I forced a vlan ID on the switch port along with the only untagged option. I thought all this behaviour might have something to do with VLANs, but connecting the interface to another router with no VLANs configured doesn't let any traffic through either.

Normally I'd say it's some kind of hardware problem (switch/cable/interface), but the interface works on the latest Fedora 38 LiveCD every time, after multiple reboots, and it's forwarding packets without any errors visible on the switch.

[Faez_Intel]

Hi drv41,

Good day and thank you for contacting Intel Community Support.

Please kindly gather the SSU logs for us to analyze further on the issue. Refer to this link below on how to gather the logs:

https://www.intel.com/content/www/us/en/support/articles/000008563/ethernet-products.html

Also, since you using the latest NVM and i40e on the Ubuntu, can you confirm the same version was used on the Fedora OS as well?

Thank you,

Faez_Intel

[drv41]

Hello Faez, thanks for getting back to me.

I'm attaching SSU output (additional interfaces are from kubernetes network) as well as well as screenshots with kernel version and basic output from (working) fedora live cd. I don't know what driver version are compiled into that kernel. I might install recent drivers on fedora, but that would require significant downtime as OS needs to be installed on HDD to test behaviour on reboots. Still can be done if you really need that info.

[Faez_Intel]

Hi drv41,

Good day and thank you for the details provided.

We are currently checking on this with our next level and we will get back to you accordingly. Please allow us some time to check on this.

If you have any other inquiries, please do not hesitate to let us know accordingly.

Thanks,

Faez_Intel

[Faez_Intel]

Hi drv41,

Good day and apologies for the delayed response from our end.

We have checked the logs and verified as well that for Ubuntu 22.04 LTS Is a supported OS for this NIC.

From the logs however, we will need a fresh logs from the non working setup for us to try and troubleshoot further on this issue as well. Please kindly help to gather a new log with the non working setup for us to check further.

Also, may I know if you are having issues when installing the driver in Ubuntu? If yes, please provide us with the exact commands that you used to install the driver.

Let us know if you have any other questions.

Thanks,

Faez_Intel

[drv41]

Hello.

- provided logs are from NON working environment. I just mentioned that it works on Fedora

- no problems installing drivers

Let me know if you need other logs.

[drv41]

Sorry, just to clarify: mbox.txt is from not working ubuntu, other screenshots was from working fedora just to prove it is working.

[drv41]

This might also be handy. You can see TX errors. This is from ubuntu 22.04 with Intel drivers installed.

[Faez_Intel]

Hi drv41,

Good day and thanks for the details provided.

As checked on the logs, we can see the driver version is not available in the logs and we are suspecting this is due to the drivers not installed properly.

Can you help to write down the command that you used to install the drivers for us to review?

Thanks,

Faez_Intel

[drv41]

Hello, I think drivers are installed.

                    Driver:"i40e"
                    Driver Path:"/lib/modules/6.2.0-26-generic/updates/drivers/net/ethernet/intel/i40e/i40e.ko"
                    Driver Provider:"Intel Corporation, <e1000-devel@lists.sourceforge.net>"
                    Driver Version:""
                    Duplex:"Unknown! (255)"
                    Firmware Version:"9.30 0x8000e5d0 1.3429.0"
                    Interface:"enp2s0f0"

install process:

cd i40e-2.23.17/src/
make
make install

output:

*** The target kernel has CONFIG_MODULE_SIG_ALL enabled, but
*** the signing key cannot be found. Module signing has been
*** disabled for this build.
make[1]: Entering directory '/usr/src/linux-headers-6.2.0-32-generic'
make[1]: Leaving directory '/usr/src/linux-headers-6.2.0-32-generic'
Installing modules...
*** The target kernel has CONFIG_MODULE_SIG_ALL enabled, but
*** the signing key cannot be found. Module signing has been
*** disabled for this build.
make[1]: Entering directory '/usr/src/linux-headers-6.2.0-32-generic'
  INSTALL /lib/modules/6.2.0-32-generic/updates/drivers/net/ethernet/intel/i40e/i40e.ko
  DEPMOD  /lib/modules/6.2.0-32-generic
make[1]: Leaving directory '/usr/src/linux-headers-6.2.0-32-generic'
make auxiliary_install
make[1]: Entering directory '/root/i40e-2.23.17/src'
Installing auxiliary...
make[1]: Leaving directory '/root/i40e-2.23.17/src'
/sbin/depmod -e -F /boot/System.map-6.2.0-32-generic  -a 6.2.0-32-generic
Updating initramfs...
update-initramfs: Generating /boot/initrd.img-6.2.0-32-generic
make mandocs_install
make[1]: Entering directory '/root/i40e-2.23.17/src'
Copying manpages...
make[1]: Leaving directory '/root/i40e-2.23.17/src'

modinfo

root@mbox:~/i40e-2.23.17/src# modinfo i40e
filename:       /lib/modules/6.2.0-32-generic/updates/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.23.17
license:        GPL
description:    Intel(R) 40-10 Gigabit Ethernet Connection Network Driver
author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion:     2C42B81353318AF07873C84
alias:          pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000158Asv*sd*bc*sc*i*
alias:          pci:v00008086d00000DDAsv*sd*bc*sc*i*
alias:          pci:v00008086d000037D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CFsv*sd*bc*sc*i*
alias:          pci:v00008086d000037CEsv*sd*bc*sc*i*
alias:          pci:v00008086d00000D58sv*sd*bc*sc*i*
alias:          pci:v00008086d00000CF8sv*sd*bc*sc*i*
alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Esv*sd*bc*sc*i*
alias:          pci:v00008086d000015FFsv*sd*bc*sc*i*
alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d0000101Fsv*sd*bc*sc*i*
alias:          pci:v00008086d00000DD2sv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:
retpoline:      Y
name:           i40e
vermagic:       6.2.0-32-generic SMP preempt mod_unload modversions
parm:           debug:Debug level (0=none,...,16=all) (int)
parm:           l4mode:L4 cloud filter mode: 0=UDP,1=TCP,2=Both,-1=Disabled(default) (int)

[Faez_Intel]

Hi drv41,

Good day.

Please kindly try to load the drivers after compiling. Below are the steps to load the drivers:

Load the module using the modprobe command.

To check the version of the driver and then load it:

 # modinfo i40e

 # modprobe i40e [parameter=port1_value,port2_value]

 Alternately, make sure that any older i40e drivers are removed from the kernel before loading the new module:

  # rmmod i40e; modprobe i40e

Please kindly try to load the drivers and let us know on the results.

Thanks,

Faez_Intel

[drv41]

Hello.

In my last message I provided you modinfo showing that module is loaded it seems. I did reboot after driver install, just to make sure. Anyway. I'm attaching output from commands you requested:

root@mbox:~# modinfo i40e
filename:       /lib/modules/6.2.0-32-generic/updates/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.23.17
license:        GPL
description:    Intel(R) 40-10 Gigabit Ethernet Connection Network Driver
author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion:     2C42B81353318AF07873C84
alias:          pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000158Asv*sd*bc*sc*i*
alias:          pci:v00008086d00000DDAsv*sd*bc*sc*i*
alias:          pci:v00008086d000037D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CFsv*sd*bc*sc*i*
alias:          pci:v00008086d000037CEsv*sd*bc*sc*i*
alias:          pci:v00008086d00000D58sv*sd*bc*sc*i*
alias:          pci:v00008086d00000CF8sv*sd*bc*sc*i*
alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Esv*sd*bc*sc*i*
alias:          pci:v00008086d000015FFsv*sd*bc*sc*i*
alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d0000101Fsv*sd*bc*sc*i*
alias:          pci:v00008086d00000DD2sv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:
retpoline:      Y
name:           i40e
vermagic:       6.2.0-32-generic SMP preempt mod_unload modversions
parm:           debug:Debug level (0=none,...,16=all) (int)
parm:           l4mode:L4 cloud filter mode: 0=UDP,1=TCP,2=Both,-1=Disabled(default) (int)
root@mbox:~# modprobe i40e
root@mbox:~# rmmod i40e; modprobe i40e
root@mbox:~# modinfo i40e
filename:       /lib/modules/6.2.0-32-generic/updates/drivers/net/ethernet/intel/i40e/i40e.ko
version:        2.23.17
license:        GPL
description:    Intel(R) 40-10 Gigabit Ethernet Connection Network Driver
author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion:     2C42B81353318AF07873C84
alias:          pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000158Asv*sd*bc*sc*i*
alias:          pci:v00008086d00000DDAsv*sd*bc*sc*i*
alias:          pci:v00008086d000037D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CFsv*sd*bc*sc*i*
alias:          pci:v00008086d000037CEsv*sd*bc*sc*i*
alias:          pci:v00008086d00000D58sv*sd*bc*sc*i*
alias:          pci:v00008086d00000CF8sv*sd*bc*sc*i*
alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Esv*sd*bc*sc*i*
alias:          pci:v00008086d000015FFsv*sd*bc*sc*i*
alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d0000101Fsv*sd*bc*sc*i*
alias:          pci:v00008086d00000DD2sv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:
retpoline:      Y
name:           i40e
vermagic:       6.2.0-32-generic SMP preempt mod_unload modversions
parm:           debug:Debug level (0=none,...,16=all) (int)
parm:           l4mode:L4 cloud filter mode: 0=UDP,1=TCP,2=Both,-1=Disabled(default) (int)

[drv41]

Hello @Faez_Intel anything else that I could check? Problem still persists. 

[drv41]

I contacted ASRockRACK and they confirmed that this problem also occurs on other boards with the same network interface. I believe this confirms that the problem is not related to my network configuration/setup.

[Faez_Intel]

Hi drv41,

Good day and apologies for the delayed response.

Thank you for the clarification. We have checked with on this issue with our engineering team and they have acknowledged on the issue. As for now, the engineering team are still working on it and there no updates as of yet for when the fix to roll out. Again, we sincerely apologize for the inconvenience.

Let us know if you have any other inquiries.

Thanks,

Faez_Intel

[Faez_Intel]

Hi drv41,

Good day. This is up follow up on a thread. Please kindly let us know if you need any further assistance for this thread.

Do let us know if you have any other questions.

Thanks,

Faez_intel

[drv41]

Hi.

Yes, working drivers would be nice. Otherwise all the money invested in the new server is a waste. And it looks like I'm not alone.

[Faez_Intel]

Hi,

Good day and apologies for the inconvenience caused.

For this issue, please kindly help to provide the output of the following command in the affected system:

ethtool -i enp2s0f0

ethtool -i enp2s0f1

Also, to ensure the drivers are actually loaded onto the system, please kindly use the below command and let us know on the output.

modprobe -r i40e; modprobe i40e

This is due to the previous command of "modinfo i40e" will only show if the i40e was only installed on the system, but does not necessarily means it will be loaded by the device.

Next, we need to confirm kernel information:

uname -r

grep "" /etc/*-release

We also need to consider link state:

ip -d link show enp2s0f0

ip -d link show enp2s0f1

Since you also made mention of "rx_dropped packets" Please kindly provide the statistic values to confirm the dropped packets:

ethtool -S enp2s0f0

ethtool -S enp2s0f1

Lastly, you have stated the following: "The interface tries to get an IP via DHCP, but mostly fails"

For this, we need to know on how you obtain the DHCP. You may let us know on what commands or mechanism used in order to obtain the DHCP

In the meantime, since there was a hardware related comment that you made regarding on the screenshot with the below errors:

i40e 0000:02:00.1: PCI-Express bandwidth available for this device may be insufficient for optimal performance.

i40e 0000:02:00.1: Please move the device to a different PCI-e link with more lanes and/or higher transfer rate.

Some PCIe x8 slots are actually configured as x4 slots. These slots have insufficient bandwidth for full line rate with dual port and quad port devices. In addition, if you put a PCIe v4.0 or v3.0-capable adapter into a PCIe v2.x slot, you cannot get full bandwidth. 

For this issue, moving your adapter to a true PCIe v3.0 x8 slot will resolve the issue for this particular error.

Thanks,

Faez_Intel

[Faez_Intel]

Hi drv41,

Good day. This is a follow up on this thread. Please kindly let us know if you need any further assistance for this thread.

Do let us know if you have any other questions.

Thanks,

 Faez_intel

[drv41]

Hello.

Thanks @Faez_Intel for your help.

I've found another person owning Asrock E3C256D4U-2L2T having exact same issues with 10G interface + problem with throughput of PCI devices (PCI-Express bandwidth available for this device may be insufficient for optimal performance). I think that there is some problem with MOBO design and it's overall functionality. I bought ASUS P12R-E-10G-2T as I was sick and tired of of going back and forth with it Asrock and it's issue. ASUS P12R-E-10G-2T has exact same NIC integrated and it works out of the box without any problems on Ubuntu 22.04. I contacted Asrock via supplier and they also confirmed problems with NIC on their mobo. I'll be returning my Asrock as it's clearly broken and who knows if this is not the same for the whole batch or series of this mobo.

[Faez_Intel]

Hi drv41,

Good day and noted on this.

Since you have used a different board and now it is working as expected, is there anything else I can assist you with for this community thread?

Kindly let us know if you have any other inquiries and we will assist you accordingly.

Best regards,

Faez.

Intel Customer Support