NIC firmware update on ROMED8-2T failed?!

[Dolpa]

Dear Folks,

I use an AsrockRack ROMED8-2T mobo with EPYC 7451 CPU and running Proxmox VE for virtualization.
This mobo has a dual Intel X550-T2 NIC onboard which has an old firmware from 2018.

The shipped NIC firmware is 1.93 80000AEE 05/25/2018.
The latest on official Intel site is 3.15 80001373 11/10/2020.

I mailed to AsrockRack support one month ago how can I update the firmware but the first question was why i want to update?
So I would like to use the SR-IOV function for KVM guest and bridge mode for LXC but this does not work at the same time with the old firmware, but will work with latest firmwares.

After a long exchange of letters with support stayed in to try the official update.
So I follow the Intel official method I used the UEFI tools.

https://downloadcenter.intel.com/download/28336/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-X550-Series

# nvmupdate64e -u -b -l -o update.xml -c nvmupdate.cfg
Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.42.7
Copyright (C) 2013 - 2020 Intel Corporation.

Config file read.
Inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Update
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Creating backup images in directory: D05099DB878F.
        Backup images created.
        Flash update started.
|======================[100%]======================|
        NVM verification started.
        Shadow RAM verification started.
|======================[100%]======================|
        Shadow RAM verification finished.
        Flash verification started.
|======================[100%]======================|
        Flash verification finished.
        NVM verification finished.
        Flash update successful.
Update security revisions
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Skipping update minimum security revisions.
Checking update availability for next tool run.
Post update inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Power Cycle is required to complete the update process.

The update.xml content is:

<?xml version="1.0" encoding="UTF-8"?>
<DeviceUpdate lang="en">
    <Instance vendor="8086" device="1563" subdevice="1563" subvendor="1849" bus="98" dev="0" func="0" PBA="000000-000" port_id="Port 1 of 2" display="Intel(R) Ethernet Controller X550-T2">
        <Module type="PXE" version="2.4.44" previous_version="2.4.32" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="EFI" version="7.8.13" previous_version="7.0.19" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="NVM" version="80001373" previous_version="80000AEE" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <VPD>
            <VPDField type="String">Intel (r) Ethernet Controller X550</VPDField>
        </VPD>
        <MACAddresses>
            <MAC address="D05099DB878F">
            </MAC>
            <AltMAC address="D05099DB878F">
            </AltMAC>
        </MACAddresses>
    </Instance>
    <Instance vendor="8086" device="1563" subdevice="1563" subvendor="1849" bus="98" dev="0" func="1" PBA="000000-000" port_id="Port 2 of 2" display="Intel(R) Ethernet Controller X550-T2">
        <Module type="PXE" version="2.4.44" previous_version="2.4.32" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="EFI" version="7.8.13" previous_version="7.0.19" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="NVM" version="80001373" previous_version="80000AEE" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <VPD>
            <VPDField type="String">Intel (r) Ethernet Controller X550</VPDField>
        </VPD>
        <MACAddresses>
            <MAC address="D05099DB8790">
            </MAC>
            <AltMAC address="D05099DB8790">
            </AltMAC>
        </MACAddresses>
    </Instance>
    <NextUpdateAvailable> 0 </NextUpdateAvailable>
    <RebootRequired> 0 </RebootRequired>
    <PowerCycleRequired> 1 </PowerCycleRequired>
</DeviceUpdate>

I did the power cycle and started the Proxmox. The NIC recognized and the linux kernel module loaded successful.

# ethtool -i enp98s0f0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80001373, 1.2203.0
expansion-rom-version:
bus-info: 0000:62:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -i enp98s0f1
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80001373, 1.2203.0
expansion-rom-version:
bus-info: 0000:62:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

However the port0 working fine BUT the port1 has no link. Both worked before update.

Now only port0 has link, I double checked the cables and switch (NETGEAR GS110EMX).

# ethtool enp98s0f0
Settings for enp98s0f0:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

# ethtool enp98s0f1
Settings for enp98s0f1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: no

I checked the firmware again and seems it's fine:

# nvmcheck64e /DEVICES
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NIC B/D/F     Ven-Dev   MAC          ENA Branding string
=== ========= ========= ============ === ======================================
1) 098/00/00 8086-1563 D05099DB878F YES Intel(R) Ethernet Controller X550-T2
2) 098/00/01 8086-1563 D05099DB8790 YES Intel(R) Ethernet Controller X550-T2

# nvmcheck64e /NIC=1 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.

# nvmcheck64e /NIC=2 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.

Intel diagnostic tool port0 and port1:

I made a backup from the old firmware, but the update tool said "Rollback blocked". So I can't rollback to the old version.

I tried firmware update on Windows10 too but same story.

Any idea? How I can revive the link? I know this is a 3rd party issue but AsrockRack said only solution is RMA and they will flash back the old firmware but I would like to avoid it because this is a long procedure for me and my system is a massive watercooling build very hard to dissassemble.

Best Regards,

Dolpa

[Caguicla_Intel]

Hello Dolpa,

Thank you for the patience on this matter.

Please see below statement for the feedback from Engineering and provide the additional details requested. 

As suspected, the difference in the Port behavior is in the MDI routing of the Port that changed. This is due to the fact that for ease of routing the MDI for a NIC, swapping the MDI order makes sense.

The before file shows the programmed NVM with MAC Addresses and the MDI set to straight.

The After file shows the usually customized fields such as the MAC Address are set to default and the MDI set to reversed.

The default for the dev starter of the new Etrack ID/EEPID is for the MDI set to reversed.

1.) Is it possible that you did not use the correct NUP package to update your current NVM on the target device?

2.) Did you use a HW programmer or one of our tools to program the dev starter NVM from the package directly to the card as is?

This is the only explanations based on the behavior described and the differences between the before and after dumps provided.

It should be noted that the NVM in the before was customized to change the value of the MDI to set it to straight as the Etrack ID for as noted above is set to reversed.

The critical value for the P1 MDI Config Field in the PMA RX Prov Port 1 Data Word is in Bit 0

0 = Straight MDI

1 = Reversed MDI

Looking forward to your reply.

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C

Intel® Customer Support

[Dolpa]

Hello Crisselle,

Thank You for the response.

As I wrote I only use the official 3.15 NUP, no other tools.

I looking at the nvupdate.cfg file and I found the EEPID for my NIC.

;**************************************************************************
; dSGVL_B0_AT2_NCSI *****************************************************
;**************************************************************************
BEGIN DEVICE
DEVICENAME: SAGEVILLE_EXT
VENDOR: 8086
DEVICE: 1563
NVM IMAGE: dSGVL_B0_AT2_NCSI_3p15.bin
SKIP OROM: FALSE
EEPID: 80001373
EEPROM MAP: dSGVL_0.txt
REPLACES: 80000526 8000059F 80000603 8000069C 800007F6 8000049A 800005D0 800005DE 800005EE 800005F8 80000634 80000651 80000688 8000069D 800007B5 8000084F 80000948 800009BA 800009E7 800009F9 80000A29 80000A73 80000AEE 80000D03 80000E3E 80000D37 80000D78 80000F49 80001112 80000F98 80000F9B 80000F9C 80001050 80001138 80001180 800011FA 8000134A
RESET TYPE: POWER
END DEVICE

So the official firmware update recognized it as an updatable NIC.

Unfortunately i don't really understand what this mdi route but can I fix this?

Best regards,

Dolpa

[Caguicla_Intel]

Hello Dolpa,

Thank you for the swift response.

We already communicated your reply to our engineers. Rest assured that we will get back to you as soon as we heard an update from them but no later than 2-4 business days.

Thank you for your kind understanding.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Dolpa,

Good day!

Please be informed that this query is still under investigation. Rest assured that we will send an update as soon as we heard from our engineers but no later than 2-4 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Dolpa]

Hello Crisselle,

Thank You, I'm waiting.

Best regards,

Dolpa

[Caguicla_Intel]

Hello Dolpa,

Good day and I hope this message finds you well! Please see below update from our engineering team. 

Do you know if you were in Debug Mode? This would allow you to write the whole NVM which reset the preserved values.

Additionally, please be informed that our engineering team is working on reproducing this issue and will take some time in our LAB. Rest assured that we will let you know once there is further updates.

Awaiting to hear from you. 

Should there be no response from you, I’ll make sure to reach out after 3 business days. 

Best regards,

Crisselle C

Intel® Customer Support

[Dolpa]

Hello Crisselle,

I don't think I was in debug mode. I just ran the update tools in UEFI mode.

Best regards,

Dolpa

[Caguicla_Intel]

Hello Dolpa,

Appreciate your swift response.

Please allow me to re-escalate this request to our engineers. 

As we also mentioned on our previous post, they are currently working on reproducing this issue and will take some time in our Lab. Rest assured that we will let give you an update as soon as we heard from them but no later than a week. 

Hoping for your kind understanding.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Dolpa

Good day!

We hope you don't mind providing the step by step procedure performed. If in case you're having trouble sharing this with us or isn't feasible on your end, feel free to let us know.

Please accept our sincerest apologies for any inconvenience caused as we are trying to find the root cause of the issue.

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you. 

Best regards,

Crisselle C

Intel® Customer Support

[Dolpa]

Hello Crisselle,

OK, here is a step by step what I did.

1. I downloaded  the NVM update utility v3.15 tool from here: https://downloadcenter.intel.com/download/28336/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-X550-Series

2. Unzipped the X550_NVMUpdatePackage_v3_15_EFI.zip content to an USB drive.

3. Rebooted the server in UEFI shell.

3. In UEFI shell I ran: nvmupdate64e -u -b -l -o update.xml -c nvmupdate.cfg.

4. Again this was the output.

# nvmupdate64e -u -b -l -o update.xml -c nvmupdate.cfg
Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.42.7
Copyright (C) 2013 - 2020 Intel Corporation.

Config file read.
Inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Update
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Creating backup images in directory: D05099DB878F.
        Backup images created.
        Flash update started.
|======================[100%]======================|
        NVM verification started.
        Shadow RAM verification started.
|======================[100%]======================|
        Shadow RAM verification finished.
        Flash verification started.
|======================[100%]======================|
        Flash verification finished.
        NVM verification finished.
        Flash update successful.
Update security revisions
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Skipping update minimum security revisions.
Checking update availability for next tool run.
Post update inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Power Cycle is required to complete the update process.

5. After the upgrade finished I did a power cycle.

That is it.

Best regards,

Dolpa

[Caguicla_Intel]

Hello Dolpa

We highly appreciate your effort in sharing the step by steps done and for your swift response.

Please allow us to forward this to our engineers. We will get back to you as soon as we heard an update from them but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Dolpa,

Good day!

Please be informed that we already communicated the step by step done on Firmware update you've shared and we are currently waiting for the feedback from our engineering team. Once we heard from them, rest assured that we will get back to you but no later than 2-4 business days.

Thank you for your kind understanding.

Best regards,

Crisselle C.

Intel® Customer Support

[Dolpa]

Hello Crisselle,

Thank You, I'm waiting for the result.

Best regards,

Dolpa

[DGuskov]

Hello!

I have exactly some problem with ASRock motherboard E3C246D4I-2T and dual X550T NIC.

Port 1 do not work after firmware update. And I can not rollback..

Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Controller        3.48(3.30)   1563 00:003 Rollback
    X550-T2                                                      Blocked

# ethtool -i eth1
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80001578, 1.1927.0
expansion-rom-version:
bus-info: 0000:03:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

I have backup files .bin and .flb

Just run  nvmupdate64 from Linux, answer Yes to make Backup and did a power cycle after  successful update to firmware version 3.30.

[DGuskov]

Hello!

This is my backup before upgrade.

[Caguicla_Intel]

Hello Dolpa,

We sincerely apologize for the delay on this matter.

Our engineers are still closely investigating this issue. We will give you an update as soon as we heard any feedback from them but no later than 2-4 business days.

Hello DGuskov,

Thank you for letting us know that you are encountering the same issue on your ASRock motherboard. Rest assured that we will report this to our engineers as well. Kindly continue to monitor this thread for any updates.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello All,

I hope this message finds you well!

Please be informed that we are still actively working on this request. Rest assured that we will give you an update as soon as we heard from our engineers but no later than 2-4 business days.

Thank you for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello All,

Good day!

We sincerely apologize for the delay as we are thoroughly checking on this matter. Please give us more time to look into this. Rest assured that we will give you an update as soon as there is any findings but not later than 2-4 business days.

Thank you for your kind understanding.

Best regards,

Crisselle C.

Intel® Customer Support

[Dolpa]

Hello Crisselle,

I think this is not the biggest problem on the earth now.  Waiting for the result.

Best regards,

Dolpa

[Caguicla_Intel]

Hello All,

Good day and I hope everything is great!

This is just an update to inform you that our engineers are still working on this request. Please expect another update within 2-4 business day.

We appreciate your patience and understanding on this matter

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello All,

Apologies for the delay on this matter.

Please be informed that we are still actively working on this issue together with our higher level Engineers to sort this out as soon as possible. We will give you an update once we've heard from them but no later than 3-5 business days.

May you all have an amazing day!

Best regards,

Crisselle C.

Intel® Customer Support