1 out of 2 10Gbe-X550-port not usable after firmware upgrade

[Frieder]

Hi,

I'm using 3 Mainboards of the type  Asrock Rack X570D4U-2L2T with Intel X550-AT2 and 2 10Gbe-Ports onboard.

To update the firmware of the Intel X550 NIC I performed the follwoing steps:

1. Rebooted the server and booted Windows 10.

2. I downloaded  the NVM update utility v3.15 tool from here: https://downloadcenter.intel.com/download/28336/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-X550-Series

2. Unzipped in windows the X550_NVMUpdatePackage_v3_15_EFI.zip 

3. Run nvmupdate64e and agreed to install the available firmware update.

After the update to the current firmware version of the Intel X550-AT2 (needed due to the virtualization features of the new firmware), only ONE of both Ethernet-Interface is getting a link.

The other interface is dead.

Firmware downgrade to the created backup (during firmware update) is not possible. It seems to be blocked due to security reasons.

The issue a appears after the first complete powerloss of the system.  Shutting down / rebooting the system via the IPMI-Interface of the Mainboard is not enough to make the issue of the firmware update visible. 

Is there already a solution for this issue?  

If helpful for you,  I can upload the backups created during firmware-flash-process of the 3 Mainboards.

If you give me support how to use nvmupdate64e to create a backup of the current firmware, I can also provide the 3 firmware-images after the firmwareupgrade.

BR

Frieder

[Caguicla_Intel]

Hello Frieder,

Thank you for posting in Intel Ethernet Communities. 

Please provide the following information that would help in checking your request.

1. What is the current Firmware version of your Intel X550-AT2?

2. When you mentioned 'The other interface is dead', does this mean that it is not detected on Device manager and there is no LED lit on the LAN port? 

3. What is the current driver version installed on your system?

4. Have you tried contacting ASRock support to request an assistance on this matter?

We'd also like to set your expectation that we may have a limited support on this query since you are using an onboard Ethernet Controller that's embedded on a third party system. For Original Equipment Manufacturer(OEM) on-board Ethernet controller, it would be best if you can also get in touch with ASRock for further support since they have customized drivers, firmware and software to enable or alter features on your system. They are in best position where you can seek assistance so you don’t lose features or customizations of the Ethernet controller.

It is also stated on Limitations and Prerequisites of the Readme file of NVM update utility(link below) that the package is intended to be used on Intel branded adapters.

https://downloadmirror.intel.com/28336/eng/readme.txt

Limitations and Prerequisites

=============================

This package is intended to be used on Intel branded adapters. Please contact

your OEM vendor for an appropriate package. In some cases this package may

update an OEM device. This package only updates the NVM image for the device

family listed on the package. Each Intel Ethernet product family has its own

NVM Update Package.

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support

[Frieder]

Hi Crisselle,

thanks for your fast reply.

1. What is the current Firmware version of your Intel X550-AT2?

Intel X550-T2 Firmware Version: 3.21 

2. When you mentioned 'The other interface is dead', does this mean that it is not detected on Device manager and there is no LED lit on the LAN port? 

There is no LED lit working on one of the 2 X550-T2-LAN Ports - even on BIOS level. 

Both X550-T2 network devices are still detected e.g. in Windows 10. 

3. What is the current driver version installed on your system?

As there is no LED lid on BIOS level on the port, and I tested the Port unsucessfully using  Windows 10 and Ubuntu Linux 20.04 i would suppose that it is not a network-driver issue. On Ubuntu Linux I'm using the drivers that come with Ubuntu Linux Kernel 5.4 - on Winows 10 the intel Driver Version 4.1.196.0.

4. Have you tried contacting ASRock support to request an assistance on this matter?

I contacted Asrock Rack Support twice (after waiting 3 weeks for an answer of my first request) without any reply until now. 

For me it looks like the same issue as reported in that thread:

https://community.intel.com/t5/Ethernet-Products/NIC-firmware-update-on-ROMED8-2T-failed/td-p/1246402/

There seems to be a general issue updating Intel network cards which are on AsRock Rack Motherboards with new firmware provided by intel.  On my side this issue affects 3 Mainboards of the type Asrock Rack X570D4U-2L2T - and in the other thread a AsRock Rack ROMED8-2T. 

Thanks for you support & BR

Frieder

[Caguicla_Intel]

Hello Frieder,

Thank you for the swift response. 

Just to clarify, can you confirm that the 3 Asrock X570D4U-2L2T boards have the same issue after the firmware update? Please also share the backups created during firmware-flash-process of the 3 Mainboards. 

Please be informed that the thread on the link you've shared is currently escalated to our engineers. Once we have the requested information above, we will escalate this request to our engineers for further investigation.

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support

[Frieder]

Hi Crisselle,

the issue, that only one Intel-X550-port is working, appears on all 3 Mainboards after the firmware upgrade.

Sadly the issue was only visible after a complete powerloss of the system. To verify the success of the first firmware upgrade we only powered off/on the server via the IPMI interface.  The result now is, that we have 3 X570D4U-2L2T Mainboards  where only one port is working.

Please find attached the backup-files created during the Intel-X550-firmware upgrade of the 3 Asrock X570D4U-2L2T Mainboards.

Thanks you so much for having a look on this issue. It might affect all owners of a a AsRock Board with X550 onboard who are performing the intel FW upgrade (e.g. to make use of virtualization functions of the new intel-nic-firmware). 

BR

Frieder

[Caguicla_Intel]

Hello Frieder,

Thank you for the effort in sharing the requested information.

Please allow us to escalate this request to our engineers and mention the other thread as a reference. Rest assured that we will get back to you as soon as possible but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello Frieder,

I hope you are doing good!

Please be informed that we already escalated this request to our engineers and they are actively working on this issue. We will get back to you as soon as we heard from them but no later than 2-4 business days.

Thank you for your kind understanding.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello Frieder,

I hope this message finds you well!

We sincerely apologize for the delay on this matter as we are thoroughly checking on this. Please give us more time to look into this query. Rest assured that we will give you an update as soon as there is any findings but not later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

[DanielBaird]

I have an ASRock Rack X570D4U-2L2T and applied the X550 firmware upgrade. I too have this exact same problem.  I'm wondering if the motherboard manufacturer is using a non-standard firmware with this X550-T2    

MB Firmware Information
BMC Firmware Version 1.10.00
BIOS Firmware Version P1.30
PSP Firmware Version 0.14.0.20
Microcode Version 0a201009

after the update only one of the two ports works. one gets no link up on the switch.   i've attached my firmware backup too. 

Num Description Ver.(hex) DevId S:B Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Controller X550-T2                               3.48(3.30) 1563 00:036 Up to date
02) Intel(R) I210 Gigabit Network Connection                      3.22(3.16) 1533 00:038 Update not available
03) Intel(R) I210 Gigabit Network Connection                      3.22(3.16) 1533 00:039 Update not available

root@gaia:/opt/Intel/APPS/NVMCHECK/Linux_x64# ./nvmcheck64e /DEVICES
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.37.01.00
QV SDK version 2.37.01.00
Copyright(C) 2012 - 2021 Intel Corporation.

NIC B/D/F Ven-Dev MAC ENA Branding string
=== ========= ========= ============ === ======================================
1) 036/00/00 8086-1563 42E3FA51E6A3 YES Intel(R) Ethernet Controller X550-T2
2) 036/00/01 8086-1563 42E3FA51E6A3 YES Intel(R) Ethernet Controller X550-T2
3) 038/00/00 8086-1533 42E3FA51E6A3 YES Intel(R) I210 Gigabit Network Connecti
4) 039/00/00 8086-1533 D05099DF90B0 YES Intel(R) I210 Gigabit Network Connecti

root@gaia:/opt/Intel/APPS/NVMCHECK/Linux_x64# ./nvmcheck64e /NIC=1 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.37.01.00
QV SDK version 2.37.01.00
Copyright(C) 2012 - 2021 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.
root@gaia:/opt/Intel/APPS/NVMCHECK/Linux_x64# ./nvmcheck64e /NIC=2 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.37.01.00
QV SDK version 2.37.01.00
Copyright(C) 2012 - 2021 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.
root@gaia:/opt/Intel/APPS/NVMCHECK/Linux_x64#

root@gaia:/opt/Intel/X550/Linux_x64# ethtool enp36s0f0
Settings for enp36s0f0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
root@gaia:/opt/Intel/X550/Linux_x64# ethtool enp36s0f1
Settings for enp36s0f1:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: no
root@gaia:/opt/Intel/X550/Linux_x64#

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.37.1.1
Copyright(C) 2013 - 2021 Intel Corporation.

./nvmupdate64e -r -m D05099DF90B2 -l output2.log

Config file read.
Inventory
[00:036:00:00]: Intel(R) Ethernet Controller X550-T2
Flash inventory started.
Shadow RAM inventory started.
Shadow RAM inventory finished.
Flash inventory finished.
OROM inventory started.
OROM inventory finished.
Update
[00:036:00:00]: Intel(R) Ethernet Controller X550-T2
Flash: Rollback blocked - skipping update.
Update security revisions
[00:036:00:00]: Intel(R) Ethernet Controller X550-T2
Skipping update minimum security revisions.

[Caguicla_Intel]

Hello Frieder,

Good day!

We sincerely apologize for the delay as we are thoroughly checking on this matter. Please give us more time to look into this. Rest assured that we will give you an update as soon as there is any findings but not later than 2-4 business days.

Thank you for your kind understanding.

Hello DanielBaird,

Thank you for letting us know that you are encountering the same issue on your ASRock motherboard. Rest assured that we will report this to our engineers as well. Kindly continue to monitor this thread for any updates.

Best regards,

Crisselle C.

Intel® Customer Support

[DanielBaird]

Thankyou Intel support. The assistance is most appreciated. I can see now that it was careless to rush ahead and apply this firmware update, as it does say in the readme as you pointed out that this is for dedicated intel cards. as a suggestion for the future, a check that warns the user that "this is an embedded chip with non standard current firmware" would be nice. 

i've logged a ticket with ASRock Rack support as well. if i hear back anything i'll report here for the benefit of others

[Caguicla_Intel]

Hello DanielBaird,

Appreciate your swift response.

Rest assured that we will forward your suggestion to our engineers. Please allow us to re-escalate this request and continue the investigation. We will get back to you as soon as possible but no later than 2-4 business days.

Best regards,

Crisselle C.

Intel® Customer Support

[Caguicla_Intel]

Hello All,

Good day!

Please be informed that this request is still under investigation. Rest assured that we will give you an update as soon as we heard from our engineers but no later than 3-5 business days. 

Thank you for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

[ArneB]

We've exactly the same problem with the onboard X550T2 of our ASRockRack X570D4U-2L2T but WITHOUT updating the firmware of the Ethernet controller. The only firmware we've updated is the BIOS from 1.20 to 1.30. The firmware of the X550T2 is still at 0x80000aee, 1.1927.0.

Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Controller       1.147(1.93)   1563 00:036 Update        
    X550-T2                                                      available
02) Intel(R) I210 Gigabit Network       3.22(3.16)   1533 00:039 Update not    
    Connection                                                   available
03) Intel(R) I210 Gigabit Network       3.22(3.16)   1533 00:040 Update not    
    Connection                                                   available

The first port works as expected, the second port is unable to link and ethtool shows unknown status:

Settings for enp36s0f0:
	Supported ports: [ TP ]
	Supported link modes:   100baseT/Full
	                        1000baseT/Full
	                        10000baseT/Full
	                        2500baseT/Full
	                        5000baseT/Full
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  100baseT/Full
	                        1000baseT/Full
	                        10000baseT/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	MDI-X: Unknown
netlink error: Operation not permitted
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes


Settings for enp36s0f1:
	Supported ports: [ TP ]
	Supported link modes:   100baseT/Full
	                        1000baseT/Full
	                        10000baseT/Full
	                        2500baseT/Full
	                        5000baseT/Full
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  100baseT/Full
	                        1000baseT/Full
	                        10000baseT/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: Unknown!
	Duplex: Unknown! (255)
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	MDI-X: Unknown
netlink error: Operation not permitted
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: no

Output of nvmcheck is normal too.

Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.37.01.00 
QV SDK version 2.37.01.00
Copyright(C) 2012 - 2021 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.

I've contacted the ASRockRack support but haven't received any answer yet.

[Caguicla_Intel]

Hello All,

How are you all doing? I hope this message finds you well!

Please accept our sincerest apologies for the delay on this matter. Kindly see below information for the update from our Engineering team.

We believe the issue is that Debug Mode is enabled which results in the single port failure following the update.

Debug Mode is entered when the signal to X550 between the Flash's SCLK pin is pulled up when the X550 powers on (like a boot strapping configuration).

This mode is for engineering and debug work only and is not supported in normal operation or basically in manufacturing.

We surmise that ASROCK is shipping the motherboards in this configuration.

The Debug Mode tells the Firmware onboard the X550 it is ok to make certain writes to the NVM it might normally. This would explain the behavior we see evidence of, and it also correlates to the isolation to this one manufacturer.

It is highly recommended to request RMA for your boards and share the aforementioned information with the OEM, ASROCK.

https://asrock.com/support/index.asp

Thank you for your utmost cooperation while we are checking your concerns. We will now proceed closing this request on our end. Your time and cooperation throughout the process is highly appreciated. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

Best regards,

Crisselle C.

Intel® Customer Support

[ArneB]

Criselle, thank you for this valuable information. I'll forward it to the ASRock Rack support. One question: Is there any way to verify that the X550 is in Debug Mode, maybe with a user space tool from Linux?

[ArneB]

I've found a simple fix for the problem. Normally we won't performing a full power off after changes to the BIOS settings. So, after enabling the second port we started the server *without* a complete power cycle of the server (BMC still running). After a complete power off to change the hardware the second 10 GBit port starts working.

It seems that the hardware needs a complete power cycle after enabling the port in the BIOS setting. Maybe this will help with the update problem too as it is stated in the README that a full power cycle is needed after certain update procedures.

[Caguicla_Intel]

Hello ArneB,

Good day and I hope this message finds you well!

Thank you for the information provided. This is indeed valuable to other users experiencing the same issue. We still highly recommend getting in touch with ASRock regarding this issue to seek for their feedback.

Regarding your additional question, 'Is there any way to verify that the X550 is in Debug Mode, maybe with a user space tool from Linux?' Please see below information for our engineering team's response.

You may need to generate logs from update process to check if preservation passed successfully. To turn on debug logs in NUT for Unix-systems: “export NUL_DEBULOG=1” and “export QV_DEBUG_LOG=0x000C0241”. The second command will generate nvmupdate.log file.

This would require access to LANCONF tool.

Also, regarding programmatically to determine if the device is in debug mode. You can, if you have Lanconf or EEUPDATE.

Current mode can be read from the Flash Access Register 0x0001001C

Bit 6 "LOCKED" is cleared in non-secure mode and set in secure mode.

If you can read the value at the word at the offset 0x0001001C, we will want the value of bit 6 specifically. 

Note: We believe that it is not DEBUG Mode, but DebugLog. Unfortunately to test this, it may result in a bricked Motherboard so we believe that ASROCK should perform this.

We would recommend to contact Asrock and provide the result. Request for replacements and do not perform the Firmware update until ASROCK determines and resolves the issue.

Feel free to let us know if you have additional questions and clarifications.

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support 

[DGuskov]

Hello!

For the second (not worked) port

0x1001C: FLA         (Flash Access)                   0x00005048

But for the  first ( worked) port

0x1001C: FLA         (Flash Access)                   0x0000504A

If I understand correctly, bit 6 is set to 1 for both ports

[Caguicla_Intel]

Hello ArneB,

Thank you for the prompt reply.

Please allow me to re-escalate this request to our engineers. We will get back to you as soon as possible but no later than 2-4 business days.

Thank you for your kind understanding.

Best regards,

Crisselle Franz

Intel Customer Support

[Caguicla_Intel]

Hello DGuskov,

Good day!

We sincerely apologize for the confusion on this matter as I addressed my reply to ArneB that we will check with engineering. 

We still highly recommend following suggestions form our engineers mentioned on my previous post to contact Asrock and provide the result. Request for replacements and do not perform the Firmware update until ASROCK determines and resolves the issue.

Please be informed that we will now close this request. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

May you have an amazing weekend!

Best regards,

Crisselle C

Intel Customer Support