NIC firmware update on ROMED8-2T failed?!

[Dolpa]

Dear Folks,

I use an AsrockRack ROMED8-2T mobo with EPYC 7451 CPU and running Proxmox VE for virtualization.
This mobo has a dual Intel X550-T2 NIC onboard which has an old firmware from 2018.

The shipped NIC firmware is 1.93 80000AEE 05/25/2018.
The latest on official Intel site is 3.15 80001373 11/10/2020.

I mailed to AsrockRack support one month ago how can I update the firmware but the first question was why i want to update?
So I would like to use the SR-IOV function for KVM guest and bridge mode for LXC but this does not work at the same time with the old firmware, but will work with latest firmwares.

After a long exchange of letters with support stayed in to try the official update.
So I follow the Intel official method I used the UEFI tools.

https://downloadcenter.intel.com/download/28336/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-X550-Series

# nvmupdate64e -u -b -l -o update.xml -c nvmupdate.cfg
Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.42.7
Copyright (C) 2013 - 2020 Intel Corporation.

Config file read.
Inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Update
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Creating backup images in directory: D05099DB878F.
        Backup images created.
        Flash update started.
|======================[100%]======================|
        NVM verification started.
        Shadow RAM verification started.
|======================[100%]======================|
        Shadow RAM verification finished.
        Flash verification started.
|======================[100%]======================|
        Flash verification finished.
        NVM verification finished.
        Flash update successful.
Update security revisions
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Skipping update minimum security revisions.
Checking update availability for next tool run.
Post update inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Power Cycle is required to complete the update process.

The update.xml content is:

<?xml version="1.0" encoding="UTF-8"?>
<DeviceUpdate lang="en">
    <Instance vendor="8086" device="1563" subdevice="1563" subvendor="1849" bus="98" dev="0" func="0" PBA="000000-000" port_id="Port 1 of 2" display="Intel(R) Ethernet Controller X550-T2">
        <Module type="PXE" version="2.4.44" previous_version="2.4.32" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="EFI" version="7.8.13" previous_version="7.0.19" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="NVM" version="80001373" previous_version="80000AEE" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <VPD>
            <VPDField type="String">Intel (r) Ethernet Controller X550</VPDField>
        </VPD>
        <MACAddresses>
            <MAC address="D05099DB878F">
            </MAC>
            <AltMAC address="D05099DB878F">
            </AltMAC>
        </MACAddresses>
    </Instance>
    <Instance vendor="8086" device="1563" subdevice="1563" subvendor="1849" bus="98" dev="0" func="1" PBA="000000-000" port_id="Port 2 of 2" display="Intel(R) Ethernet Controller X550-T2">
        <Module type="PXE" version="2.4.44" previous_version="2.4.32" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="EFI" version="7.8.13" previous_version="7.0.19" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="NVM" version="80001373" previous_version="80000AEE" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <VPD>
            <VPDField type="String">Intel (r) Ethernet Controller X550</VPDField>
        </VPD>
        <MACAddresses>
            <MAC address="D05099DB8790">
            </MAC>
            <AltMAC address="D05099DB8790">
            </AltMAC>
        </MACAddresses>
    </Instance>
    <NextUpdateAvailable> 0 </NextUpdateAvailable>
    <RebootRequired> 0 </RebootRequired>
    <PowerCycleRequired> 1 </PowerCycleRequired>
</DeviceUpdate>

I did the power cycle and started the Proxmox. The NIC recognized and the linux kernel module loaded successful.

# ethtool -i enp98s0f0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80001373, 1.2203.0
expansion-rom-version:
bus-info: 0000:62:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -i enp98s0f1
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80001373, 1.2203.0
expansion-rom-version:
bus-info: 0000:62:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

However the port0 working fine BUT the port1 has no link. Both worked before update.

Now only port0 has link, I double checked the cables and switch (NETGEAR GS110EMX).

# ethtool enp98s0f0
Settings for enp98s0f0:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

# ethtool enp98s0f1
Settings for enp98s0f1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: no

I checked the firmware again and seems it's fine:

# nvmcheck64e /DEVICES
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NIC B/D/F     Ven-Dev   MAC          ENA Branding string
=== ========= ========= ============ === ======================================
1) 098/00/00 8086-1563 D05099DB878F YES Intel(R) Ethernet Controller X550-T2
2) 098/00/01 8086-1563 D05099DB8790 YES Intel(R) Ethernet Controller X550-T2

# nvmcheck64e /NIC=1 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.

# nvmcheck64e /NIC=2 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.

Intel diagnostic tool port0 and port1:

I made a backup from the old firmware, but the update tool said "Rollback blocked". So I can't rollback to the old version.

I tried firmware update on Windows10 too but same story.

Any idea? How I can revive the link? I know this is a 3rd party issue but AsrockRack said only solution is RMA and they will flash back the old firmware but I would like to avoid it because this is a long procedure for me and my system is a massive watercooling build very hard to dissassemble.

Best Regards,

Dolpa

[Caguicla_Intel · Accepted solution]

Hello All,

How are you all doing? I hope this message finds you well!

Please accept our sincerest apologies for the delay on this matter. Kindly see below information for the update from our Engineering team.

We believe the issue is that Debug Mode is enabled which results in the single port failure following the update.

Debug Mode is entered when the signal to X550 between the Flash's SCLK pin is pulled up when the X550 powers on (like a boot strapping configuration).

This mode is for engineering and debug work only and is not supported in normal operation or basically in manufacturing.

We surmise that ASROCK is shipping the motherboards in this configuration.

The Debug Mode tells the Firmware onboard the X550 it is ok to make certain writes to the NVM it might normally. This would explain the behavior we see evidence of, and it also correlates to the isolation to this one manufacturer.

It is highly recommended to request RMA to your boards and share the aforementioned information with the OEM, ASROCK.

https://asrock.com/support/index.asp

Thank you for your utmost cooperation while we are checking your concerns. We will now proceed closing this request on our end. Your time and cooperation throughout the process is highly appreciated. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

Best regards,

Crisselle C.

Intel® Customer Support

[Dolpa]

Hello Crisselle,

Thank you for your hard work and the final solution.  Seems it will be RMA. I will share this information with Asrock.

Best regards,

Dolpa

[Caguicla_Intel]

Hello Dolpa,

You're most welcome. It's been my pleasure helping you out!

Please feel free to post a new question if you may need additional assistance from Intel Ethernet team in the future. We will now proceed closing this request.

Thank you for choosing Intel and may you have a great rest of your week!

Best regards,

Crisselle C.

Intel® Customer Support