Intel 10G NIC no longer works in Debian 11 after Firmware update

[MattW-UW]

Hi,

I've got an Intel 10G NIC in a Dell R640 server running Debian 11 - it was working fine until a recent firmware update and now I can no longer get an LACP bond up and running on it.

The NOC says that it doesnt see any IP address or even a MAC address on the port that is connected

I ran the System Support Utility and will attach that here

Note that there is also a 1G interface on this card (i350/i520 motherboard daughter card) and the 1G NIC seem to be working fine still

I did try rolling back the firmware to the prior version but that didnt seem to work

I would appreciate any suggestions how to debug this or what else I can try.

Thank you,

Matt W 

[Caguicla_Intel]

Hello Matt W,

Thank you for posting in Intel Ethernet Communities. 

Please provide the information below that would help in checking your request. 

1. When you mentioned Firmware update, is it Firmware update on the Ethernet card or your motherboard?

2. We'd like to inform you that Debian OS is not listed as supported Operating System for Intel X710 card. You may visit link below for list of Supported Operating Systems for Retail Intel® Ethernet Adapters

https://www.intel.sg/content/www/xa/en/support/articles/000025890/ethernet-products.html

Awaiting to your reply. 

We will follow up after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C.

Intel® Customer Support

[MattW-UW]

I think that both the Dell R640 motherboard firmware recently updated, and also the Intel NIC firmware.

Note that there are 3 different NIC's in this machine -all intel, all different:

root@ika:~# lspci | grep -i net
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
19:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
19:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
3b:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
3b:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01)
root@ika:~#

Here is the motherboard BIOS detail from "dmidecode":

BIOS Information
Vendor: Dell Inc.
Version: 2.15.1
Release Date: 06/15/2022

The motherboard "Network Daughter Card" is the primary interface of concern at the moment, here is the firmware detail for that rNDC:

	Intel(R) Gigabit 4P X520/I350 rNDC - 24:6E:96:05:65:04	Multiple versions	19.5.12
		Components Current Available Intel(R) Gigabit 4P X520/I350 rNDC - 24:6E:96:05:65:04 20.5.15 19.5.12 Intel(R) Gigabit 4P X520/I350 rNDC - 24:6E:96:05:65:05 20.5.15 19.5.12 Intel(R) Ethernet 10G 4P X520/I350 rNDC - 24:6E:96:05:65:00 20.5.15 19.5.12 Intel(R) Ethernet 10G 4P X520/I350 rNDC - 24:6E:96:05:65:02 20.5.15 19.5.12

[Caguicla_Intel]

Hello Matt W,

Thank you for the swift response.

We will be glad to further check this request with our engineers. Prior to that, will you be able to provide front and back photos of the adapters for us to check if you are using an Original Equipment Manufacturer(OEM) or retail version of Intel Ethernet Adapter. This will help us with the investigation. 

We will also be looking for the YottaMark* or BradyID* Stickers as these are authentication labels for Intel® Ethernet Adapters. Refer to link below for YottaMark* and BradyID* Stickers on Intel® Ethernet Adapters

https://www.intel.com/content/www/us/en/support/articles/000007074/ethernet-products/gigabit-ethernet-adapters-up-to-2-5gbe.html

Looking forward to your reply. 

We will reach out after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C.

Intel Customer Support

[MattW-UW]

I've not got photos of all the different intel adapter that have problems, but here are some Dell NDC adapters that are problematic

[MattW-UW]

heres my dell embedded nic

[Caguicla_Intel]

Hello MattW-UW,

Thank you for the reply. 

Just to confirm, can you share how many Intel adapters are problematic on your setup? Are they all PCIe adapters and not embedded NIC? Can you list the models of the adapters on this thread and how many of them have issues.

Awaiting to your reply. 

We will follow up after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello MattW-UW,

I hope this message finds you well!

This is just a follow up to check how many Intel adapters are problematic on your setup? Are they all PCIe adapters and not embedded NIC? Can you list the models of the adapters on this thread and how many of them have issues.

Looking forward to your reply. 

We will reach out after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C.

Intel® Customer Support

[MattW-UW · Accepted solution]

Dear Intel Community & support folks

This turned out to not be an Intel or server drive firmware issue after all

Once we did a Juniper EX4550 switch reboot, things starting looking way better almost immediately.

Overview of all the things we tried:
- changing twinax to singlemode fiber
- scrutinizing fiber optic transceivers to make sure each matched hardware vendor (jupiter / intel / dell)
- replacing fiber cables
- pouring over bug report on the debian and ubuntu bug tracker for ixgbe bugs
- reading cranky forum posts about x710 NIC weirdness
- Support calls to Dell
- Dell Server Update Utility (.ISO)
- Dell System Update
- Pulling out x710 network daughter card from mobo and using just the PCI card
- Wondering why systems that had no x710 card (just x520) were also sometimes working, sometimes not with the exact same drivers and firmware rev
- moving stuff to 1G ports as available (for some reason those worked reliably well on same switch!)

- Cursing the Dell C6320 because it only has 2x 10G ports and one USB port - so couldnt try 1G or plug a USB key into it to load firmware updates when no network connectivity
- iDRAC ISO mounting SUU to do those updates instead
- Installing an alternate, virtualized temporary server so customers could reach *something *
- Ordered Broadcom replacement daughterboard/NICS for some boxes

We have been aggressively taking Dell firmware updates via the OME virtual appliance and pushing them to our PowerEdge systems for several years now. I really believed that was the problem (a mismatch between OS drivers and firmware)  and was operating for far too long under this incorrect assumption.

We have a fair bit of cleanup and testing still to do but things are looking much better now with a reboot of the switch.

"Have you tried turning it off and on again?" prevails once again

Case closed and thank you for your attention to this issue

-Matt Weatherford

University of Washington

Seattle, WA

[Caguicla_Intel]

Hello Matt,

Thank you for the update and sharing the issue resolution on this thread. 

We are glad to hear that the issue is now resolved. We will now proceed closing this request as per your advise. We appreciate your time and effort throughout the process. Please feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

Have an amazing weekend ahead and stay safe!

Best regards,

Crisselle C.

Intel® Customer Support