E810-XXV Firmware Upgrade Error - Can not upgrade to NVM 4.10

[azaidi]

Hello,

I am writing to report an issue that I am experiencing with the nvmupdate script on the Intel E810-XXV and E810-C devices.

Currently, we are on NVM/firmware 4.0 and are attempting to upgrade to 4.10 to align with the 27.7 Software release.

I am using the E810_NVMUpdatePackage_v4_10_Linux.tar.gz update package, which was obtained from the link provided on your website:

-----

NVM package used : E810_NVMUpdatePackage_v4_10_Linux.tar.gz

Link : [https://www.intel.com/content/www/us/en/download/19626/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapters-e810-series-linux.html]

-----

When executing the update script, here is the output I am getting: 

-----

NVM upgrade output (NVM upgrade log attached with debug mode)

# ./nvmupdate64e

Intel(R) Ethernet NVM Update Tool

NVMUpdate version 1.39.20.2

Copyright(C) 2013 - 2022 Intel Corporation.

WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.

Inventory in progress. Please wait [|.........]

Num Description                          Ver.(hex)  DevId S:B    Status

=== ================================== ============ ===== ====== ==============

01) Intel(R) Ethernet 25G 2P E810-XXV   4.00(4.00)   159B 00:049 Update not

    OCP                                                          available

02) Intel(R) Ethernet 100G 2P E810-C    4.00(4.00)   1592 00:075 Update not

    Adapter                                                      available

03) Intel(R) Ethernet 100G 2P E810-C    4.00(4.00)   1592 00:152 Update not

    Adapter                                                      available

Tool execution completed with the following status: Device not found.

Press any key to exit.

-----

Please see below for the details about the devices in question:

-----

# lspci -vvv | grep E810

31:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)

        Subsystem: Intel Corporation Ethernet 25G 2P E810-XXV OCP

                Product Name: E810-XXV 25GbE Controller

31:00.1 Ethernet controller: Intel Corporation Ethernet Controller E810-XXV for SFP (rev 02)

        Subsystem: Intel Corporation Ethernet 25G 2P E810-XXV OCP

                Product Name: E810-XXV 25GbE Controller

4b:00.0 Ethernet controller: Intel Corporation Ethernet Controller E810-C for QSFP (rev 02)

        Subsystem: Intel Corporation Ethernet 100G 2P E810-C Adapter

                Product Name: E810-C 100GbE Controller

# ethtool -i sriov0

driver: ice

version: 1.10.1.2

firmware-version: 4.00 0x800139bc 21.5.9

expansion-rom-version:

bus-info: 0000:4b:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes



Device info detail  

*-network:0       

description: Ethernet interface       

product: Ethernet Controller E810-C for QSFP       

vendor: Intel Corporation       

physical id: 0       

bus info: pci@0000:98:00.0       

logical name: sriov2       

version: 02       

serial: 40:a6:b7:83:de:90       

width: 64 bits       

clock: 33MHz       

capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre       

configuration: autonegotiation=off broadcast=yes driver=ice driverversion=1.10.1.2 duplex=full

firmware=4.00 0x800139bc 21.5.9 latency=0 link=yes multicast=yes port=fibre       
resources: irq:18 memory:d4000000-d5ffffff memory:d8010000-d801ffff memory:d1000000-d10fffff memory:d7000000-d7ffffff memory:d8220000-d841ffff

-----

Below are my system specs: 

OS: Rocky Linux 8.6

I have also attached the nvmupdate log as a part of this ticket for further analysis. Based on the log, the update is unable to occur due to a missing QV driver. We have installed the iqv-1.2.0.21 driver and are still experiencing the same issue. In addition to the QV driver, we have also installed the following drivers, based on the Intel Compatibility matrix: 

ICE-1.10.1.2

IAVF-4.6.1

IRDMA-1.10.15

Unfortunately, we have not been able to successfully update the firmware as of yet. Any help would be much appreciated.

Thank you
Arsalan

[Faez_Intel]

Hi Arsalan,

Thank you for reaching out to Intel Customer Support.

Thank you as well for providing the details of your current issue. Based on this, please kindly help to provide the details as below:

1) Please kindly help to gather the SSU logs for us to analyze further on this issue. Refer to this link on how to gather the SSU logs:

https://www.intel.com/content/www/us/en/support/articles/000008563/ethernet-products.html

2) Is the E810 faced any issue when you are running with the current setup, firmware and the current OS?

Best regards,

Faez.

Intel Customer Support 

[Faez_Intel]

Hi Arsalan,

Good day and I hope this message finds you well!

This is just a follow up regarding the information we requested for us to proceed with checking on your query.

Do feel free to inquire us if you have any other questions that you may have and we will gladly assist you accordingly.

Best regards,

Faez.

Intel Customer Support

[azaidi]

Hi Faez, 

I apologize for the delay. We will provide the necessary information here to you shortly. Please keep this ticket open! We appreciate your help.

Thank you
Arsalan

[azaidi]

Hi Faez,

Please see below for the requested SSU logs.

Additionally, we have not noticed any issues with the E810 with the current setup.

Please let us know if there is any additional information needed from our end.

Thank you
Arsalan

[Faez_Intel]

Hi Arsalan,

Good day and thanks for the response.

Thanks for the SSU logs provided. It seems that you are using a DELL R750 system.

For this case, NICs that came together from the OEM's manufacturer, we would advise to use the firmware provided by the OEM themselves as they have configured the firmware accordingly to the system.

For Dell E810 latest firmware, refer to this link: https://www.dell.com/support/home/en-sg/drivers/driversdetails?driverid=25ffj&oscode=naa&productcode=poweredge-r750

As for the NVM update, you may refer to this link as well, page 19: https://downloadmirror.intel.com/769278/Release_Notes_28.0.pdf

The NVM update process will need Power On Reset on PTP devices unless the user manually turns off PTP activity before the NVM update process is started.

— As a user with root privileges, run the following on each PF: ethtool --set-priv-flags ethX

ptp_wt_enabled off

— The driver will log the following message in dmesg: ICE_FLAG_PTP_WT_ENABLED is disabled, PTP workthread will not continue

— When the PTP threads are disabled, the NVM update process should work as usual. To reenable the PTP threads again, the ice driver reload is necessary, or a platform reboot. 

Please ensure that the PTP is turned on and you may try to perform the NVM update again. Let me know if you need any clarifications.

Thanks,

Faez

[azaidi]

Hi Faez,

We have updated the firmware to the latest version (21.5.9) and are still experiencing issues when attempting to upgrade.

Specifically, we do not see the option to set the ptp flag within the ethtool command. Please see below the system outputs when executing the recommended commands:

----

[root@rocky-cu-wk01 ~]# ethtool --set-priv-flags sriov0 ptp_wt_enabled off

netlink error: bit name not found (offset 48)
netlink error: Operation not supported
[root@rocky-cu-wk01 ~]#
[root@rocky-cu-wk01 ~]#
[root@rocky-cu-wk01 ~]# ethtool -i sriov0
driver: ice
version: 1.10.1.2
firmware-version: 4.00 0x800139bc 21.5.9
expansion-rom-version:
bus-info: 0000:4b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[root@rocky-cu-wk01 ~]# ethtool --show-priv-flags sriov0

Private flags for sriov0:
link-down-on-close : off
fw-lldp-agent : off
channel-inline-fd-mark : off
channel-pkt-inspect-optimize : on
channel-pkt-clean-bp-stop : off
channel-pkt-clean-bp-stop-cfg: off
vf-true-promisc-support : off
mdd-auto-reset-vf : off
vf-vlan-pruning : off
legacy-rx : off
cgu_fast_lock : off
dpll_monitor : off

extts_filter : off
allow-no-fec-modules-in-auto : off

----

Please see below for the dmesg log while trying to perform the upgrade:

----

[root@rocky-cu-wk01 ~]# dmesg -T | grep -i error
[Thu Feb 23 18:06:36 2023] ERST: Error Record Serialization Table (ERST) support is initialized.
[Thu Feb 23 18:06:50 2023] ACPI Error: No handler for Region [SYSI] (00000000edac3840) [IPMI] (20210604/evregion-135)
[Thu Feb 23 18:06:50 2023] ACPI Error: Region IPMI (ID=7) has no handler (20210604/exfldio-265)
[Thu Feb 23 18:06:50 2023] ACPI Error: Aborting method _SB.PMI0._GHL due to previous error (AE_NOT_EXIST) (20210604/psparse-531)
[Thu Feb 23 18:06:50 2023] ACPI Error: Aborting method _SB.PMI0._PMC due to previous error (AE_NOT_EXIST) (20210604/psparse-531)
[Thu Feb 23 18:06:50 2023] ACPI Error: AE_NOT_EXIST, Evaluating _PMC (20210604/power_meter-759)
[Thu Feb 23 18:07:08 2023] ice 0000:31:00.1 eno12409: Error creating RoCEv2 filter

----

Please advise on next steps. 

Additionally, just as a sanity check, we would just like to confirm if what we are trying to accomplish here is possible. Our goal is to update our NVM/firmware from 4.0 to 4.10 to align with the software release 27.7. 

Looking forward to your response.

Thank you

Arsalan

[Faez_Intel]

Hi Arsalan,

Good day and I hope this message finds you well!

Thank you for the detailed description after performing the suggested action. For this matter, please allow us some time to confirm on the compatibility level between NVM 4.10 and the latest software releases. We will update you soon once we have confirmed this from our end.

Let us know if you have any other clarifications.

Best regards,

Faez.

Intel Customer Support 

[Faez_Intel]

Hi Arsalan,

Good day and I hope this message finds you well!

From the ethtool, it seems that the card is an OEM card. Can you help to confirm with us if your card is from an OEM manufacturer or did you purchased the card directly from Intel?

This is due to card that are OEM manufactured is not supported by Intel NVM. For OEM cards, you will need to get the NVM update from the manufacturer's website.

Best regards,

Faez.

Intel Customer Support 

[Faez_Intel]

Hi Arsalan,

Good day and I hope this message finds you well!

This is a follow up on the thread. Kindly let us know if you have any further inquiries or assistance needed.

Should there be no response from you, I’ll make sure to reach out on.

Best regards,

Faez.

Intel Customer Support

[Faez_Intel]

Hi Arsalan,

I hope you're having a wonderful day. I am just sending a follow up to check if you still have questions or clarifications. We will wait for your reply. As for now, we will be closing your thread.

If you want to continue support, please reply to this community thread and we will reopen your case or create a new one so that we can continue to support you. 

Best regards,

Faez.

Intel Customer Support