1 out of 2 10Gbe-X550-port not usable after firmware upgrade

[Frieder]

Hi,

I'm using 3 Mainboards of the type  Asrock Rack X570D4U-2L2T with Intel X550-AT2 and 2 10Gbe-Ports onboard.

To update the firmware of the Intel X550 NIC I performed the follwoing steps:

1. Rebooted the server and booted Windows 10.

2. I downloaded  the NVM update utility v3.15 tool from here: https://downloadcenter.intel.com/download/28336/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-X550-Series

2. Unzipped in windows the X550_NVMUpdatePackage_v3_15_EFI.zip 

3. Run nvmupdate64e and agreed to install the available firmware update.

After the update to the current firmware version of the Intel X550-AT2 (needed due to the virtualization features of the new firmware), only ONE of both Ethernet-Interface is getting a link.

The other interface is dead.

Firmware downgrade to the created backup (during firmware update) is not possible. It seems to be blocked due to security reasons.

The issue a appears after the first complete powerloss of the system.  Shutting down / rebooting the system via the IPMI-Interface of the Mainboard is not enough to make the issue of the firmware update visible. 

Is there already a solution for this issue?  

If helpful for you,  I can upload the backups created during firmware-flash-process of the 3 Mainboards.

If you give me support how to use nvmupdate64e to create a backup of the current firmware, I can also provide the 3 firmware-images after the firmwareupgrade.

BR

Frieder

[DGuskov]

Hello!

My problem with the E3C246D4I-2T motherboard was solved by the firmware from the Asrock  - https://download.asrock.com/TSD/nvmupdate(3.15).zip

There is only EFI update package but the point is the ASRR_X550_0_0_002_0.bin file and nvmupdate.cfg - these files can be used with another flasher. I use it with NVMUpdatePackage_550_Series_v3_15.zip from Intel.

Power cycle after upgrade is required.

If you use linux - add the "iomem=relaxed" to the GRUB_CMDLINE_LINUX, update grub and reboot.

[thomas001le]

Hi!

I have exactly the same board as you have. I tried the firmware in the link, but had no luck flashing it, even from an EFI shell. The output I get is only "Update not available". Maybe it is because I am on stock intel firmware 3.30 and the asrock image seems to be based on 3.15. Did you do anything special to allow downgrading of the firmware version or were you just on 3.15 and not 3.30 when you applied the update?

Thanks!

[DGuskov]

I upgraded to 3.30 first, then downgraded to version 3.15 from Asrok.

You need modify nvmupdate.cfg

Find sections with string "NVM IMAGE: ASRR_X550_0_0_002_0.bin"

to string started with "REPLACES:" add 80001578 at the end.

add string "IMAGE DOWNGRADE: TRUE" before "RESET TYPE"

Power cycle with disconnection of all power cables is mandatory.

This is updated nvmupdate.cfg in the attach, I was flashing our MB with this configuration file.

[thomas001le]

Thanks, it worked like a charm! 

[Octave]

Hi,

I am in the same situation (with ROMED8-2T) but I upgraded to the latest firmware from Intel which is 3.60. I used your modified nvmupdate.cfg and added the ASRR_X550_0_0_002_0.bin from Asrock to the folder, but I get this (picture attached), when running "nvmupdate64e.efi  -c nvmupdate.cfg".

Any tips? Thank you!

[Octave]

Found the solution! Since I was on 3.60 the new EEPID was 8000172D, so I had to replace 80001578 by 8000172D in the nvmupdate.cfg and now it works!

[doroseracledes]

I had the same problem with a AsRockRack motherboard SPC621B8.

Already installed unreleased CPLD, BIOS and BMC firmware versions from AsRockRack but still had issues with the 3 X550-T2 installed.

Went with the Intel 3.60 firmware and after that one of the port on each NIC stopped working.

I downloaded the AsRock 3.15 firmware from the above link https://download.asrock.com/TSD/nvmupdate(3.15).zip but replacing 80001578 with 8000172D in "REPLACES" in nvmupdate.cfg didn't work for me, it would just not detect the installed NICs so here's what I did:

I found all the EEPIDs from the intel 3.60 firmware:

```

~$ egrep EEPID Downloads/X550/EFI2x64/nvmupdate.cfg
EEPID: 80001726
EEPID: 80001727
EEPID: 80001728
EEPID: 80001729
EEPID: 8000172A
EEPID: 8000172B
EEPID: 8000172C
EEPID: 8000172D
EEPID: 8000172E
EEPID: 8000172F
EEPID: 80001730
EEPID: 80001731
EEPID: 80001732
EEPID: 80001733
EEPID: 80001734
EEPID: 80001735

```

and added them all to the nvmupdate.cfg file like this

```

~$ more Downloads/X550_ASROCK/nvmupdate/nvmupdate.cfg
CURRENT FAMILY: 1.0.0
CONFIG VERSION: 1.14.0

;**************************************************************************
; dSGVL_B0_AT2_NCSI *****************************************************
;**************************************************************************
BEGIN DEVICE
DEVICENAME: SAGEVILLE_EXT
VENDOR: 8086
DEVICE: 1563
NVM IMAGE: ASRR_X550_0_0_002_0.bin
SKIP OROM: FALSE
EEPID: 80001373
EEPROM MAP: dSGVL_0.txt
REPLACES: 80000526 8000059F 80000603 8000069C 800007F6 8000049A 800005D0 800005DE 800005EE 800005F8 80000634 80000651 80000688 8000069D 800007B5 8000084F 80000948 800009BA 800009E7 800009F9 80000A29 80000A73 80000AEE 80000D03 80000E3E 80000D37 80000D78 80000F49 80001112 80000F98 80000F9B 80000F9C 80001050 80001138 80001180 800011FA 8000134A 80001726 80001727 80001728 80001729 8000172A 8000172B 8000172C 8000172D 8000172E 8000172F 80001730 80001731 80001732 80001733 80001734 80001735
IMAGE DOWNGRADE: TRUE
RESET TYPE: POWER
END DEVICE

```

after that I managed to downgrade from 3.60 to 3.15:

X550 downgrade output

and even though I got an error when I rebooted the machine all 3 X550-T2s where working fine

I hope this helps someone else

[Ramalama]

@Frieder Thank you for starting this Topic!

@DGuskov Thank you a lot for the solution, you saved my butt

@EveryoneElse Thanks guys for digging Intel, that we got a solution!

I thought my second port is not working at all, because since i bought the board, i never needed the second port.

Seems like in the meantime, i've updated my X550 Adapter, just forgot about it.

My board is out of warranty and the Asrock Rack X570d4i-2t costs now 631€ as cheapest, i bought it for 413€ back in 2020.

So there was no hope for me for any RMA at all!

However, i found this topic, flashed the X550 adapter to the asrock rom (3.60 back to 3.15) and my second adapter works!!!

Im so happy, because that saved me literally 631€.

@doroseracledes 

I have a question to you either.

Since there is absolutely no Changelog, for any Rom updates...

What have changed since 3.15?

Because i literally still have some bugs, especially if you bind a Linux Bridge (vmbr) to the PF and you passthrough the VF to a guest, you can't talk from the Guest(VF) to anything on the Linux Bridge that is bound to the PF.

There is a workaround to modify fdb tables, but it's a workaround.

However, is it one of those bugfixes between 3.15 and 3.60 ?

Cheers!

[doroseracledes]

Hi @Ramalama 

This post helped me and that's why I added additional information on how I got my cards to work. I'm glad it helped you too!

I did some googling for changelog and found this post that contains a link to a PDF with the features supported on every firmware version:

https://community.intel.com/t5/Ethernet-Products/X550-NVM-Firmware-changelog/m-p/677527

the PDF link

https://www.intel.com/content/dam/www/public/us/en/documents/release-notes/ethernet-controller-x550-feature-support-matrix.pdf

I hope this helps

Best

Doros

[Ramalama]

Thank you very much!
Im stupid, i wanted to write that question to the guy from intel and wanted to thank you for digging

But then i wrote to you about the changelog:-)

However, thanks a lot!
What im seeing from the thread you linked is that there is absolutely nothing changed between 3.15 and 3.60 other as the number and the changelog is a mystery hole.

Only the driver version requirements changes, so i believe that we can find a changelog at least at the corresponding open source driver versions (kernel basically).

Thanks again!