LACP not working with linux ixgbe 10G NIC and Ubuntu 22, starts working only after driver reload

[Joonas]

I have a couple of HP DL380 servers with Intel network cards. The cards are connected to HP 25G switches using 10G copper transceivers and configured with LACP active-active for a 20G connection. This setup works fine with Ubuntu 20.04. With a newer release Ubuntu 22.04 though, only 1 NIC port stays up after reboot, the other one is in DOWN state and LACP is using only one NIC port instead of two.

I've tried to debug the issue for some time and I noticed that with Ubuntu  22.04, after I do:

rmmod ixgbe && modprobe ixgbe

The second NIC comes up and LACP starts working in active-active mode. Since reloading the driver fixes the problem, I suspect that the issue is within this driver.  The LACP configuration is identical between Ubuntu releases.

Network card:

04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
DeviceName: Embedded FlexibleLOM 1 Port 1
Subsystem: Hewlett-Packard Company Ethernet 10Gb 2-port 560FLR-SFP+ Adapter
Kernel driver in use: ixgbe
Kernel modules: ixgbe

Driver version:

ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k

Logs and command outputs in the attached text

Many thanks.

[IntelSupport]

Hello Joonas,

Thank you for posting on the Intel communities.

Please give us 2-3 business day to check on this and get back to you.

In the meantime please go thru below link as it might help understanding what you needed as well. You can also find the latest driver and kindly make sure you have the updated driver. If you have updated the driver please check if the LACP started to work or not.

https://www.intel.com/content/www/us/en/support/articles/000005688/ethernet-products.html

Kindly provide us with an update as well once you have done above steps.

Thank you.

Regards,

Hayat

Intel Customer Support

[IntelSupport]

Hello Joonas,

Regarding your thread, we are following up to find out if you were able to complete the actions, we previously recommended.

Please reply to this thread to confirm, so we can continue helping with a resolution. Looking forward to receiving your reply!

Thank you.

Regards,

Hayat

Intel® Customer Support

[Joonas]

Hi Hayat,

I found the source package for ixgbe 5.19.6 through your link. It has pre-compiled modules only for RHEL and SLES though, not for Ubuntu. I downloaded the package on Ubuntu 22.04.2 LTS, installed the kernel source package but was unable to build the module. The error messages go beyond my understanding as I'm not a kernel developer:

make[1]: Entering directory '/usr/src/linux-headers-5.15.0-88-generic'
CC [M] /home/ubuntu/ixgbe-5.19.6/src/ixgbe_main.o
In file included from ./include/linux/export.h:45,
from ./include/linux/linkage.h:7,
from ./include/linux/kernel.h:8,
from ./include/linux/list.h:9,
from ./include/linux/module.h:12,
from /home/ubuntu/ixgbe-5.19.6/src/ixgbe_main.c:8:
/home/ubuntu/ixgbe-5.19.6/src/ixgbe_main.c: In function ‘ixgbe_atr’:
/home/ubuntu/ixgbe-5.19.6/src/ixgbe_main.c:9872:30: error: ‘VXLAN_HEADROOM’ undeclared (first use in this function); did you mean ‘VXLAN_HF_RCO’?
9872 | VXLAN_HEADROOM))
| ^~~~~~~~~~~~~~
./include/linux/compiler.h:78:45: note: in definition of macro ‘unlikely’
78 | # define unlikely(x) __builtin_expect(!!(x), 0)
| ^
/home/ubuntu/ixgbe-5.19.6/src/ixgbe_main.c:9872:30: note: each undeclared identifier is reported only once for each function it appears in
9872 | VXLAN_HEADROOM))
| ^~~~~~~~~~~~~~
./include/linux/compiler.h:78:45: note: in definition of macro ‘unlikely’
78 | # define unlikely(x) __builtin_expect(!!(x), 0)
| ^
make[2]: *** [scripts/Makefile.build:297: /home/ubuntu/ixgbe-5.19.6/src/ixgbe_main.o] Error 1
make[1]: *** [Makefile:1909: /home/ubuntu/ixgbe-5.19.6/src] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-88-generic'
make: *** [Makefile:100: default] Error 2

So I was not able to test the new release.

[IntelSupport]

Hello Joonas,

Kindly go thru this readme files. - latest version

*The ixgbevf driver can only be activated on kernels that support SR-IOV. It supports all of the above devices except for the Intel(R) Ethernet Controller 82598*

Does the kernel you use support SR-IOV?

If the latest version is not applicable for Ubuntu, please use this version instead Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections under Linux*

Kindly go thru the readme files and release notes thoroughly before using above version.

Kindly let us know if you have any questions.

Regards,

Hayat

Intel Customer Support

[IntelSupport]

Hello Joonas,

Regarding your thread, we are following up to find out if you were able to complete the actions, we previously recommended.

Please reply to this thread to confirm, so we can continue helping with a resolution. Looking forward to receiving your reply!

 Thank you.

Regards,

Hayat

Intel® Customer Support

[IntelSupport]

Hello Joonas,

Regarding your thread, we are following up to find out if you were able to complete the actions, we previously recommended.

Please reply to this thread to confirm, so we can continue helping with a resolution. Looking forward to receiving your reply!

Kindly let us know if you have any further questions.

Else if none, we will proceed with closure.

Thank you.

Regards,

Hayat

Intel® Customer Support

[Joonas]

Hi,

I found the solution to compiling the driver from this thread:

https://community.intel.com/t5/Ethernet-Products/Unable-to-compile-driver-xgbe-5-19-6-on-Debian-11-Bullseye/td-p/1518628

After changing the ixgbe_main.c it compiled successfully. I think you should create a new release for new kernels based on this.

I'll return to the original issue in the next reply.

[Joonas]

After updating to the newest ixgbe driver version, I can confirm that the behaviour stays the same as described in my first post.

After reboot, 1 out of 2 10Gbe ports are down (log messages are the same as my original post). After doing rmmod ixgbe && modprobe ixgbe, both ports are up and stay up.

[IntelSupport]

Hi Joonas,

I am glad that both port are up and the similar thread had manage to provide some assistance in your issue.

If all is good, I will proceed to close the case.

Once the case is closed it will no longer be monitored by Intel.

Kindly open a new case if you have further questions.

Regards,

Hayat

Intel Customer Support

[Joonas]

Hi,

Please read carefully. The issue is NOT resolved.

After reboot, the other interface is down. Only after a workaround (rmmod, modprobe) it comes up. This is not good behaviour and needs to be fixed.

[Joonas]

Hello?