X710 for 10GBASE-T failed firmware update

[tdr_]

One system here has 2 X710 10GB ethernet ports.

Unfortunately they are on the motherboard and not on a pcie card.   The vendor of the board provided a firmware update for the NICs under their product downloads and reported to support Linux.  The download was a shell script wrapper around the nvmupdate64e tool and executed it to a nvmupdate.cfg file that they provided firmware files with it, too. 

The inventory happened fine but the flashing failed.  The shell script also failed to include execution of nvmupdate.cfg -b to backup the current firmware.    Presently the BIOS shows N/A for one MAC address while the other card shows one. 

When booted into Linux, of course neither port is usable, , dmesg reports this (followed by a stack trace):

[ 4.518397] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[ 4.518400] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
[ 4.519074] i40e 0000:06:00.0: Firmware recovery mode detected. Limiting functionality.
[ 4.519077] i40e 0000:06:00.0: Refer to the Intel(R) Ethernet Adapters and Devices User Guide for details on firmware recovery mode.
[ 4.522513] i40e 0000:06:00.0: fw 8.6.68629 api 1.15 nvm 9.40 0x8000f09e 1.3122.0 [8086:15ff] [1043:881b]

[ 4.944383] i40e 0000:06:00.0: eeprom check failed (-5), Tx/Rx traffic disabled

[ 4.958102] i40e 0000:06:00.0: MSI-X vector reservation failed: -34
[ 4.958103] i40e 0000:06:00.0: MSI-X not available, trying MSI

The firmware tool (direct Intel download today) reports it cannot create inventory and ethtool 

./nvmupdate64e -i -l inv.log

Config file will not be read.
Warning: Cannot initialize port: [00:006:00:00] Intel(R) Ethernet Controller X710 for 10GBASE-T
Warning: Cannot initialize port: [00:006:00:01] Intel(R) Ethernet Controller XL710 Generic ID
Warning: Cannot initialize port: [00:006:00:02] Intel(R) Ethernet Controller XL710 Generic ID
Warning: Cannot initialize port: [00:006:00:03] Intel(R) Ethernet Controller XL710 Generic ID

I was able to get an RMA for this once, during which they replaced the board.  When returned, it too had the old firmware version which is listed as having vulnerabilities, so I tri,ed their packaging of the tool on it and it happened again.  Asus has been less than helpful since despite being under warranty and it being their provided tool that failed to take the preventative backup step and then failed during the update.

Is there any hope of recovering the devices without an original dump of the firmware to recover from?

[pujeeth]

Hello tdr_,

Thank you for posting in the Intel Community. We sincerely apologize for the delay in our response. We have received your concern and would like to assure you that assisting you is our top priority.

Could you kindly let us know if the issue has been resolved, or if you require further assistance from us? Your prompt response will greatly help us in diagnosing and resolving the issue as quickly as possible.

We appreciate your patience and understanding, and we look forward to hearing from you soon.

Regards

Pujeeth_Intel

[tdr_]

Thanks for your response.

The problem "reverted" to a previous state to where the NIC firmware is out of date but they at least work.

Late December I purchased an a second add-on gpu card.  Because of cabling, I moved the exist Arc A770 to another slot when I added the RTX 4070 super.   The GPU upgrade went fine and the system posted/booted as expected.  

I then checked for bios/efi updates on the Asus and pulled their release from dated 2024/11/15 and used the BMC interface to apply the update..  The bios/uiefi update process resets all the efi config options, including re-enabling the on-board X710 network cards I had left disabled since they were not operational.   When I booted to uefi setting to re-disable them, both NIC ports showed a MAC address.   Previous UEFI resets and refreshing did not ever recover the card MAC addresses, so this was "new."

 Querying the interfaces using ethanol from Linux, I see they are back to their original NIC firmware:

    # ethtool -i eno2np1
    driver: i40e
    version: 6.13.2-gentoo-12-feb-2025
    firmware-version: 8.60 0x8000d4c1 1.3122.0
    expansion-rom-version:
    bus-info: 0000:06:00.1
    supports-statistics: yes
    supports-test: yes
    supports-eeprom-access: yes
    supports-register-dump: yes
    supports-priv-flags: yes

I am not sure whether moving PCI devices around somehow reset the network cards to a working state or if applying the BIOS update corrected the problem.  The BIOS update did not report having any network card firmware in it, but it did contain management engine firmware 16.11.17.1871.  

Asus still has the same network card firmware version 9.4 available, but I'm hesitant to attempt flashing it again using their provided tooling.  It does not archive existing  firmware and failed on two motherboards for me when using their bundled nvmupdate64e and their config file.   Asus does not provide an efi bootable version of their update like Intel does and they were very unable to support Linux use of the tool despite providing it for Linux use.  I've copied the updater contents below.  The listed  version to replace does match the ethanol output, but previously it also matched but then failed and was not able to recover.

The Asus shell script to flash contains:

    ** note that -b is still not part of their process, so a backup of NVM is not taken

    # cat ./asus/nvmupdate.cfg

    chmod +x nvmupdate64e
    ./nvmupdate64e -i -l -c nvmupdate.cfg
    ./nvmupdate64e -u -l -c nvmupdate.cfg

The bundled nvmupdate.cfg by Asus:

    CURRENT FAMILY: 1.0.0
    CONFIG VERSION: 1.14.0

    ;Release 19.3/19.4 to Release 20.0

    ;X710-TM CRVL
    BEGIN DEVICE
    DEVICENAME: XL710
    VENDOR: 8086
    DEVICE: 15FF
    NVM IMAGE: ETA_SWAP_X710-AT2_9p40_CFGID14p3_NCSI_1.04_8000F09E.bin
    OROM IMAGE: BootIMG.FLB
    EEPID: 8000F09E
    REPLACES: 8000D4C1
    RESET TYPE: REBOOT
    END DEVICE

    ; nvmupdate.cfg generated on 07/28/2023, 10:55:37

[pujeeth]

Hello tdr_,

Thank you for contacting Intel.

This is the first follow-up regarding the issue you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided.

Feel free to reply to this email, and we'll be more than happy to assist you further.

Regards,

Pujeeth

Intel Customer Support Technician

[pujeeth]

Hello tdr_,

Greetings!

We will proceed to close this case. If you find that you still required assistance, we kindly request you to respond to the case. This will allow us to either reopen the current case or initiate a new one.

Regards

Pujeeth_Intel

[tdr_]

"We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided."

 I did not find a plan of action attached.

The upstream (from Intel) firmware tool when used to scan my cards did not recognize the devices as upgradable by it (probably due to OEM / motherboard vendor tagging somewhere).  I can re-open a case with Asus, but I'm extremely hesitant to use their tooling since it bricked the onboard X710 for 10GBASE-T interfaces on two of the same motherboard models previously and they don't follow a best practice of archiving in-place firmware before attempting to flash + they seem ill-equipped to support Linux at all, despite offering Linux users the tooling.   I am presently still running vulnerable firmware on the network cards. 

[pujeeth]

Hello tdr_,

If you purchased an OEM-branded system( ASUS) or appliance, please contact your OEM for a software driver version compatible with your specific firmware version.

Using software and firmware versions that have not been validated to work together may impact performance, feature availability, and system stability.

Would request you to reach out to OEM for further assistance.

Regards

Pujeeth_Intel