nvmupdate64e access error for XL710 after failed NVM upgrade

mscastanho · ‎12-08-2023

Hi, I have an XL710-QDA2 running on Ubuntu 22.04 (HWE kernel, 6.2.0-35-generic) using the default Linux i40e driver.

The NIC was running a very old firmware version (4.24 0x800013fc 0.0.0 as indicated by ethtool) so I tried upgrading to the latest version 9.30 using the nvmupdate64e (downloaded from [1]). The upgrade ended with an error and since then the NIC is unusable and running the tool again shows Access Error.

# ./nvmupdate64e 

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.39.56.8
Copyright(C) 2013 - 2023 Intel Corporation.


WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [*.........]


Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Converged           N/A(N/A)   1583 00:007 Access error  
    Network Adapter XL710-Q2                                                   


Tool execution completed with the following status: Device not found.
Press any key to exit.

I tried rolling back the firmware using the -r and -m flags and the backup image created by the upgrade tool, but without luck so far.

# ls 6805CA306848/
6805CA306848.bin  6805CA306848.flb

# ./nvmupdate64e -r -m 6805CA306848 

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.39.56.8
Copyright(C) 2013 - 2023 Intel Corporation.

# echo $?
19

According to [2] and [3], I have an Intel card, not an OEM, as the Vendor/Device ID matches the XL710QDA2 ID from [2].

# lspci -nn | grep XL710
07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ [8086:1583] (rev 01)
07:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ [8086:1583] (rev 01)

Is there any hope to recover this NIC? How can I rollback the original firmware or install a newer one? Thanks in advance.

Further info:

Driver messages on kernel logs:

# dmesg | grep i40e
[    8.010328] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[    8.043618] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
[   20.516125] i40e 0000:07:00.0: PF reset failed: -15
[   20.516837] i40e 0000:07:00.0: Entering recovery mode due to repeated FW resets. This may take several minutes. Refer to the Intel(R) Ethernet Adapters and Devices User Guide.
[   20.521852] i40e: probe of 0000:07:00.0 failed with error -15
[   32.180003] i40e 0000:07:00.1: PF reset failed: -15
[   32.180838] i40e 0000:07:00.1: Entering recovery mode due to repeated FW resets. This may take several minutes. Refer to the Intel(R) Ethernet Adapters and Devices User Guide.
[   32.185692] i40e: probe of 0000:07:00.1 failed with error -15

Output of lspci:

# lspci -vvv -d 8086:1583
[157428.273296] pci 0000:07:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update
07:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
        Physical Slot: 2
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 135
        NUMA node: 0
        Region 0: Memory at f5000000 (64-bit, prefetchable) [size=8M]
        Region 3: Memory at f6800000 (64-bit, prefetchable) [size=32K]
        Expansion ROM at f2100000 [virtual] [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable- Count=129 Masked-
                Vector table: BAR=3 offset=00000000
:[157428.409293] pci 0000:07:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update
                PBA: BAR=3 offset=00001000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x8 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
ductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [e0] Vital Product Data
                Not readable
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 48-68-30-ff-ff-ca-05-68
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 16, stride: 1, Device ID: 154c
                Supported Page Size: 00000553, System Page Size: 00000001
[157534.633351] pci 0000:07:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update
                Region 0: Memory at 00000000f6000000 (64-bit, prefetchable)
                Region 3: Memory at 00000000f6810000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
                Device specific mode supported
                No steering table available
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [1d0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel modules: i40e
07:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter XL710-Q2
        Physical Slot: 2
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 135
        NUMA node: 0
        Region 0: Memory at f5800000 (64-bit, prefetchable) [size=8M]
        Region 3: Memory at f6808000 (64-bit, prefetchable) [size=32K]
        Expansion ROM at f2180000 [virtual] [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable- Count=129 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00001000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 4096 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x8 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [e0] Vital Product Data
                Not readable
                Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP- SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 48-68-30-ff-ff-ca-05-68
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
                VF offset: 79, stride: 1, Device ID: 154c
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000f6400000 (64-bit, prefetchable)
                Region 3: Memory at 00000000f6910000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
                Device specific mode supported
                No steering table available
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel modules: i40e

[1] https://www.intel.com.br/content/www/br/pt/download/18190/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-700-series.html

[2] https://www.intel.com/content/www/us/en/support/articles/000056278/ethernet-products/legacy-ethernet-products.html

[3] https://www.intel.com/content/www/us/en/support/articles/000005612/ethernet-products/gigabit-ethernet-adapters-up-to-2-5gbe.html

Yogaeasvaran · ‎12-11-2023

Hello mscastanho,

Greetings for the day!

Thank you for contacting Intel Customer Support.

We do understand that your Ethernet XL710-QDA2 having trouble to update the firmware.

Since you have tried several steps with unsuccessful, you would need to update the correct version of the firmware.

Below is the link for you to follow step by step to ensure the updating process is correct with driver version as well.

Link: https://www.intel.com/content/www/us/en/support/articles/000030492/ethernet-products.html

Updating the firmware could cause damage to the card if the wrong firmware is applied.

Please do let us know if once you tried.

Regards,

Yogaeasvaran

mscastanho · ‎12-12-2023

Hi @Yogaeasvaran ,

Those are actually the steps I followed in the first place during the update that caused the issue. I tried again:

- Downloaded NVM update utility version 9.30

- Copied the Linux version to the server

- Ran nvmupdate64e

But the issue is the same:

# ./nvmupdate64e 

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.39.56.8
Copyright(C) 2013 - 2023 Intel Corporation.


WARNING: To avoid damage to your device, do not stop the update or reboot or power off the system during this update.
Inventory in progress. Please wait [*.........]


Num Description                          Ver.(hex)  DevId S:B    Status
=== ================================== ============ ===== ====== ==============
01) Intel(R) Ethernet Converged           N/A(N/A)   1583 00:007 Access error  
    Network Adapter XL710-Q2                                                   


Tool execution completed with the following status: Device not found.
Press any key to exit.

Should I use an specific version of the NVM update utility considering that I was using 4.24 before the failed upgrade?

Yogaeasvaran · ‎12-12-2023

Hello mscastanho,

Greetings for the day!

Thank you for response.

Yes, you may try update to update to the version before failed and check.

Please do let us know if once you tried as we may isolate this further.

Regards,

Yogaeasvaran

mscastanho · ‎12-12-2023

I was unable to find the 4.24 version for download at [1], so I tried with version 4.42. The error is slightly different now, but still no success.

# ./nvmupdate64e 

Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.24.33.08
Copyright (C) 2013 - 2015 Intel Corporation.


WARNING: TO AVOID DAMAGE TO YOUR DEVICE, DO NOT EXIT OR REBOOT OR POWER OFF THE SYSTEM DURING THIS UPDATE
Inventory in progress. Please wait [*-........]


Num Description                            Device-Id B:D   Adapter Status
=== ====================================== ========= ===== ====================
01) Intel(R) Ethernet Converged Network Ad 8086-1583 07:00 Access error        


Tool execution completed with the following status: Communication with base driver failed. Please verify that the base driver is present

[1] https://www.intel.com/content/www/us/en/download/18190/25222/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-700-series.html

Yogaeasvaran · ‎12-12-2023

Greetings for the day!

Thank you for your response.

You have tried to update the firmware but unsuccessful and not able to be solved upon trying the steps I've given.

Could you please provide me the SSU log to isolate this issue further.

Please follow the guide from below link:

Help Guide for the Intel® System Support Utility

Regards,

Yogaeasvaran

mscastanho · ‎12-12-2023

The SSU log is attached.

There were also some error messages when I ran the script:

# ./ssu.sh 
This product is not supported on this operating system.
     Would you like to try to scan? (y/n)
y
/bin/cat: /sys/devices/virtual/dmi/id/board_vendor: No such file or directory
Cannot get driver information: No such device

Yogaeasvaran · ‎12-12-2023

Greetings for the day!

Thank you for your response.

We are checking this issue and will get back you the soonest.

Regards,

Yogaeasvaran

Yogaeasvaran · ‎12-13-2023

Greetings for the day!

Thank you for your patience.

Could you please provide us NIC PBA/Part number and share a picture of the NIC. Also, please verify the place of purchase for the NIC.

Regards,

Yogaeasvaran

mscastanho · ‎12-14-2023

I'm trying to find a way to take a picture of the NIC, since this is a remote server. I'll get back to you as soon as possible.

But I also share part of the sentiment of the previous comment, most of the similar threads I found follow the same pattern.

Let's hope there's a solution for this one.

itinc · ‎12-13-2023

Was Intel able to fix this issue, it seems like every post that has the same issue gets the same response.

"Send a picture and that's not a recognized Intel adapter!"

Do issues actually get fixed??

This can't be impossible. It's a failed update, that should have a work around after Intel's utility fails, are they just trying to sell more cards?

Yogaeasvaran · ‎12-14-2023

Greetings for the day!

Thank you for your feedback and we will wait for the Picture and details from you.

Regards,

Yogaeasvaran

itinc · ‎12-15-2023

Here are the images from my board.

Yogaeasvaran · ‎12-16-2023

Greetings,

Thank your feedback with details.

We will check with our engineering team to isolate further and will get back you the soonest.

Regards,

Yogaeasvaran

Yogaeasvaran · ‎12-19-2023

Greetings,

Thank you for the patience.

As per our engineering team advice, the Feature Support matrix shows that it is necessary to update to 4.42 before going to higher versions.

https://www.intel.com/content/www/us/en/support/articles/000093197/ethernet-products.html

Please try to perform recovery state following the link:

https://www.intel.com/content/www/us/en/support/articles/000089896/ethernet-products.html

We also share the Quick usage for your guidance.

https://www.intel.com/content/www/us/en/support/articles/000088453/ethernet-products.html

Please do ket us know if need anything else.

Regards,

Yogaeasvaran

mscastanho · ‎12-19-2023

So according to the recovery guide I need to run the NVM Update tool to get things fixed, but doesn't it make we go back to square 1? Since the update tool is not working properly anymore with the card.

As I mentioned previously, I also tried to use the NVM Update tool version 4.42, but it failed with the following message:

Tool execution completed with the following status: Communication with base driver failed. Please verify that the base driver is present

Looks like the tool is not working properly with the upstream Linux i40e driver that is installed in the machine.

So I'm trying to install the i40e driver from Intel, version 1.2.37, which according to the support matrix is compatible with the 4.42 firmware version.

I'll update the thread when I get more info.

Yogaeasvaran · ‎12-19-2023

Greetings,

Thank you for the feedback.

We will be waiting for your update.

Regards,

Yogaeasvaran

Yogaeasvaran · ‎12-22-2023

Greetings,

We would like to follow up if you able to get details could you please let us know when you can be able to provide feedback.

Regards,

Yogaeasvaran

Yogaeasvaran · ‎12-25-2023

Greetings,

We would like to follow up 2nd time if you able to get details could you please let us know when you can be able to provide feedback.

Regards,

Yogaeasvaran

Yogaeasvaran · ‎12-28-2023

Greetings,

Kindly note that as we have not received any response from our previous follow-ups, we will be closing this request.

If you have any more questions in the future, please don't hesitate to post a new question, as this thread will no longer be monitored.

Regards,

Yogaeasvaran

sanjay_bwk · ‎09-11-2024

Hi,

I am facing the same situation, I tried updating the firmware using the latest 9.52 version from here: https://www.intel.com/content/www/us/en/download/18190/non-volatile-memory-nvm-update-utility-for-intel-ethernet-network-adapter-700-series.html

The NVM update failed, and now I am not able to use the NIC. I tried the above steps to use the 4.42 version but getting this error:

Tool execution completed with the following status: Communication with base driver failed. Please verify that the base driver is present

With other versions the NVM update tool is showing Access error.

I am using this on Ubuntu 24.04.