ixgbe fails to properly recognize dual Onboard X540-AT2

[Prashant_Gupta]

I am having issues with the ixgbe driver in Linux (specifically Ubuntu 22.04).  The machine is a Supermicro server with a X9DRW-CTF31 motherboard. It has TWO built in X540-AT2 10GBase-T ethernet ports.   On boot, one of the onboard interfaces is labeled as disabled.  I have replicated this on CentOS 8 and Ubuntu 22.04.  The ixgbe module must be unloaded and re-loaded for both interfaces to be recognized.  Supermicro has no firmware updates specific to the ethernet interfaces, and I have the latest BIOS and firmware updates for the board. These ports are built-in to the motherboard so I am at a loss as to why this is happening. They are connected to a DELL PowerConnect 6248P switch.  It should eb noted that bonding is set up for both ports, but this of course fails when the machine is booted because one of the interfaces fails to come up.  This is resolved with re-loading of the ixgbe driver. Both NICs are recognized by lspci, and there is obviously not a hardware issue since the ports BOTH come up when the driver is reloaded. I could write a script to unload and reload the driver after boot but I was wondering if there is a fix for the driver.

Below is an output and commands of the scenario starting from just after boot:

root@object2:/home/openstack# dmesg | grep ixgbe

[    6.319419] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    6.336281] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    6.653176] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 40, Tx Queue count = 40 XDP Queue count = 0
[    6.785232] ixgbe 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    7.239249] ixgbe 0000:01:00.0: MAC: 3, PHY: 0, PBA No: 010000-000
[    7.239255] ixgbe 0000:01:00.0: 0c:c4:7a:63:72:b0
[    7.386345] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[    8.313781] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 40, Tx Queue count = 40 XDP Queue count = 0
[    8.397234] ixgbe 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    8.421225] ixgbe 0000:01:00.1: MAC: 3, PHY: 0, PBA No: 010000-000
[    8.421239] ixgbe 0000:01:00.1: 0c:c4:7a:63:72:b1
[    8.567247] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[    8.568742] ixgbe 0000:01:00.0 eno1: renamed from eth0
[   23.915048] ixgbe 0000:01:00.0: registered PHC device on eno1
[   27.294552] ixgbe 0000:01:00.0 eno1: NIC Link is Up 1 Gbps, Flow Control: RX/TX
root@object2:/home/openstack# lshw -C network
  *-network:0               
       description: Ethernet interface
       product: Ethernet Controller 10-Gigabit X540-AT2
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: eno1
       version: 01
       serial: aa:7d:21:e6:5a:a2
       size: 1Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 100bt-fd 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=5.15.0-107-generic duplex=full firmware=0x800003e2 latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
       resources: irq:80 memory:dd200000-dd3fffff ioport:8020(size=32) memory:dd404000-dd407fff memory:90000000-900fffff memory:90100000-901fffff
  *-network:1 DISABLED
       description: Ethernet interface
       product: Ethernet Controller 10-Gigabit X540-AT2
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:01:00.1
       logical name: eth1
       version: 01
       serial: 0c:c4:7a:63:72:b1
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 100bt-fd 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=5.15.0-107-generic firmware=0x800003e2 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:156 memory:dd000000-dd1fffff ioport:8000(size=32) memory:dd400000-dd403fff memory:90200000-902fffff memory:90300000-903fffff
  

root@object2:/home/openstack# rmmod ixgbe && modprobe ixgbe

root@object2:/home/openstack# dmesg | grep ixgbe
[    6.319419] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[    6.336281] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    6.653176] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 40, Tx Queue count = 40 XDP Queue count = 0
[    6.785232] ixgbe 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    7.239249] ixgbe 0000:01:00.0: MAC: 3, PHY: 0, PBA No: 010000-000
[    7.239255] ixgbe 0000:01:00.0: 0c:c4:7a:63:72:b0
[    7.386345] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[    8.313781] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 40, Tx Queue count = 40 XDP Queue count = 0
[    8.397234] ixgbe 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[    8.421225] ixgbe 0000:01:00.1: MAC: 3, PHY: 0, PBA No: 010000-000
[    8.421239] ixgbe 0000:01:00.1: 0c:c4:7a:63:72:b1
[    8.567247] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[    8.568742] ixgbe 0000:01:00.0 eno1: renamed from eth0
[   23.915048] ixgbe 0000:01:00.0: registered PHC device on eno1
[   27.294552] ixgbe 0000:01:00.0 eno1: NIC Link is Up 1 Gbps, Flow Control: RX/TX
[  312.668876] ixgbe 0000:01:00.1: complete
[  312.669957] ixgbe 0000:01:00.0: removed PHC on eno1
[  313.069087] ixgbe 0000:01:00.0: complete
[  313.130485] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[  313.130488] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[  313.426650] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 40, Tx Queue count = 40 XDP Queue count = 0
[  313.510437] ixgbe 0000:01:00.0: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[  313.534452] ixgbe 0000:01:00.0: MAC: 3, PHY: 0, PBA No: 010000-000
[  313.534458] ixgbe 0000:01:00.0: 0c:c4:7a:63:72:b0
[  313.673425] ixgbe 0000:01:00.0 eno1: renamed from eth0
[  313.712815] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[  313.929891] ixgbe 0000:01:00.0: registered PHC device on eno1
[  314.010628] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 40, Tx Queue count = 40 XDP Queue count = 0
[  314.094433] ixgbe 0000:01:00.1: 32.000 Gb/s available PCIe bandwidth (5.0 GT/s PCIe x8 link)
[  314.118465] ixgbe 0000:01:00.1: MAC: 3, PHY: 0, PBA No: 010000-000
[  314.118471] ixgbe 0000:01:00.1: 0c:c4:7a:63:72:b1
[  314.453854] ixgbe 0000:01:00.1: registered PHC device on eth0
[  314.592212] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[  317.289355] ixgbe 0000:01:00.0 eno1: NIC Link is Up 1 Gbps, Flow Control: RX/TX
[  319.304055] ixgbe 0000:01:00.1 eth0: NIC Link is Up 1 Gbps, Flow Control: RX/TX
root@object2:/home/openstack# lshw -C network
  *-network:0               
       description: Ethernet interface
       product: Ethernet Controller 10-Gigabit X540-AT2
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: eno1
       version: 01
       serial: 0c:c4:7a:63:72:b0
       size: 1Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 100bt-fd 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=5.15.0-107-generic duplex=full firmware=0x800003e2 latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
       resources: irq:80 memory:dd200000-dd3fffff ioport:8020(size=32) memory:dd404000-dd407fff memory:90000000-900fffff memory:90100000-901fffff
  *-network:1
       description: Ethernet interface
       product: Ethernet Controller 10-Gigabit X540-AT2
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:01:00.1
       logical name: eth0
       version: 01
       serial: 0c:c4:7a:63:72:b0
       size: 1Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 100bt-fd 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=5.15.0-107-generic duplex=full firmware=0x800003e2 latency=0 link=yes multicast=yes port=twisted pair slave=yes speed=1Gbit/s
       resources: irq:156 memory:dd000000-dd1fffff ioport:8000(size=32) memory:dd400000-dd403fff memory:90200000-902fffff memory:90300000-903fffff
 
root@object2:/home/openstack#

As you can see, the second onboard X540-AT2 interface is disabled until i run "rmmod ixgbe && modprobe ixgbe", then it is enabled and functions normally.  Any help would be greatly appreciated.

FYI - I have replicated this issue on all four of my X9DRW-CTF31 machines, and all of them have X540-AT2 hardware, with multiple versions/flavors of linux.  The problem remains the same with the ixgbe driver. Is there a firmware or driver fix for this?

Prashant

[Sazirah]

Hi Prashant_Gupta,

Thank you for contacting Intel Ethernet Communities.

Please be inform that the current OS used (Linux Ubuntu 22.04) is not supported as per Supermicro regardless the OS is supported by the Intel X540-AT2.

We advise you to use the OS version from your OEM as they have configured the ethernet card to fit their own product.

Intel has our own generic driver version however we do not recommend you to use since the ethernet card is built in the Supermicro motherboard. Therefore, we kindly advise you to check with the OEM Supermicro for further info regarding this issue since they are more expert in providing support for this product.

Unfortunately we are unable to extend support for this product, as it falls outside of our support boundaries.

Thank you for using Intel products and services.

Regards,

Sazzy_Intel

[Prashant_Gupta]

Thank you for your response.  Supermicro no longer supports this board, and there is no firmware for these per them.  I know the X540-AT2 chipset is used in many onboard components, but if I am not mistaken, it was used in standalone cards as well. You may not be able to answer this but would you know if these onboard type X540-AT2 NICs have firmware that can be flashed?  Your website DOES have firmware listed on the X540-AT2 support page.  Lastly, have you or any of your colleagues seen this issue with INTEL branded X540-AT2 chipsets on standalone PCIe NICs?  It might provide some insight into what might be going on.  I know you do not support these boards/NICs but perhaps your branded NICs in the same family might have similar issues.  FYI I have used the lower versions of Redhat (CentOS) specified by Supermicro and these problems remain.  No problems using Windows however, which tells me this is not really a hardware issue but an ixgbe driver issue.  This is why I asked the question in this forum. This is not a hardware problem, but a software driver issue with your chipset.  Any insight you can give would be appreciated, but I understand if you cannot assist me.  If I purchase a standalone intel branded X540-AT2 and have problems with the ixgbe driver in linux would I be eligible for help with this issue?  Thank you again.

Prashant

[Sazirah]

Hi Prashant_Gupta,

Thank you for your reply.

Regarding this, let me check from my end and will get back to you soon with any update.

Thank you for your cooperation.

Regards,

Sazzy_Intel

[Prashant_Gupta]

I await your response

[Subhashish]

Hi Prashant_Gupta,

Greetings!

Yes, there is no issue with the hardware as per you description of the issue.

As we were reviewing your last post, we came across "Your website DOES have firmware listed on the X540-AT2 support page".

As far as I am aware, Intel only will provide drivers for its product. Firmware will be only provided by the OEM as Firmware will made in consideration to the OEM system and will differ from different Manufacturer.

Please see below link, You will only find drivers but not firmware:

https://www.intel.com/content/www/us/en/products/sku/60020/intel-ethernet-controller-x540at2/downloads.html

Hence the Manufacturer of the Board is only one to provide the firmware.

Aside, please see the below link :-

https://www.intel.com/content/www/us/en/download/14302/intel-network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-connections-under-linux.html

Please try the latest driver 5.19.9, Please also cross check in your board manufacturer's support page, if there is any new driver release available.

Do note that the product in question is discontinued by intel as well. So we may or may not have further updates on the already existing drivers for X540 AT2.

Regards,

Subhashish_Intel.

[Prashant_Gupta]

My apologies. You are correct.  The firmware was for the X550 series boards. I will try to compile the driver, although I thought the linux kernel for ubuntu shipped with the latest ixgbe?  Is this not the case?  Also, do you have any .deb packages or only RPM?  I can of course compile but was wondering if you had any links to premade install files for other distributions.  Thank you for your help with this.

[Sazirah]

Hi Prashant_Gupta,

Please note that we only have RPM version as stated in the website ( https://www.intel.com/content/www/us/en/download/14302/intel-network-adapter-driver-for-pcie-intel-10-gigabit-ethernet-network-connections-under-linux.html) that is tested and supported by Intel.

We strongly advise you to use the OS version from your OEM as they have configured the ethernet card to fit their own product. For any change of different configuration may impact the performance.

Also, please also cross check in your board manufacturer's support page, if there is any new driver release available. Do note that the product in question is discontinued by intel as well. So we may not have further updates on the already existing drivers for X540 AT2.

Regards,

Sazzy_Intel

[Sazirah]

Hi Prashant_Gupta,

Good day,

This is related to your enquiries above, I would like to check if everything is working fine at your end?

If you have any questions or clarifications, please let me know.

Regards,

Sazzy_Intel

[Prashant_Gupta]

I will be at the machine next week likely and will update you on the process.  It is possible this new driver might do the trick.  You have been most helpful, and I am most grateful.

Prashant

[Sazirah]

Hi Prashant_Gupta,

Thank you for your reply.

Sure this is noted. We will be waiting for your update next week.

Regards,

Sazzy_Intel

[Prashant_Gupta]

I am unable to compile the driver.

"fatal error: linux/pci-aspm.h: No such file or directory #include <linux/pci-aspm.h>"

I saw some posts regarding this in your forums, but did not see a resolution. It seems the newer kernels do not have this file and the source will not compile.  Any ideas?

I have including a link to the thread below.

Prashant

Possible workaround for installing latest ixgbe drivers on Ubuntu 20? - Intel Community

[Sazirah]

Hi Prashant_Gupta,

According to the Supermicro OS compatibility list, we found that the server did not support the latest ubuntu either 20.04 or 22.04 . You may want to refer to link below for the OS compatibility for your own product.

https://www.supermicro.com/support/resources/OS/C602J.cfm

We advise you to check and verify this with your OEM Supermicro as they will be able to provide the suitable resolution for this issue.

Regards,

Sazzy_Intel

[Sazirah]

Hi Prashant_Gupta,

Good day.

I would like to follow up with you regarding issue above. May I know if there is any assistance required from our end? As advised previously, please use the origin OS that Supermicro offer, or you can contact Supermicro support to check and verify on this issue as they will be more expert on this.

Regards,

Sazzy_Intel

[Sazirah]

Hi Prashant_Gupta,

Good day.

This is regarding your enquiry above.

Since we have not seen an update, we will be closing this forum case from our end. If you need any additional information, please submit a new question as this thread will no longer be monitored.

Regards,

Sazzy_Intel