Community
cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
Beginner
407 Views

Issue with HP DL380 Gen10 Server and xxv710-2 Intel Card

We have the same issue as https://forums.intel.com/s/question/0D50P00004OMCUdSAP/issue-with-hp-dl380-gen10-server-with-intel-x...

 

We have two xxv710-2 cards in slots 4 and 6, HP reports that the fimware on the cards is 1.1747.0. We have two servers in different locations, one being the "main" server and the other as a failover in case the other fails).

 

The servers have RedHat Virtualisation (latest version) and runs a highly tuned Fortinet firewall as a VM (using 4 VF's, two from each card). The fortigates are not setup as HA, but are managed and the routing is changed when the manager observes that a server goes down. We are using one as the "main" server and since being setup in this manner, this one stays up, with the "fallback" server falling over at no particular predictable time. The servers have been swapped over at a point, and the one being the "main" server does not exhibit the failure (i.e. it is always the "fallback" that has the issue.

 

The symptoms we have is that the VM's are turned off and the server resets itself and reboots. We can see the same errors as in the linked post. As far as the OS goes we see no errors as the server has reset (i.e. powered off).

 

I came across the attached post whilst trying to find the latest drivers, the HP SSP tool does not have any higher firmware for the intel card.

 

Any suggestions, is the firmware the latest as the numbers I have don't seem to relate to any numbers on the Intel Support site. Everything was built 2 months back with the latest and greatest versions.

 

Paul

0 Kudos
40 Replies
Highlighted
87 Views

Hello Paul,

 

Thank you for posting in Intel Ethernet Communities. 

 

Please provide the following details for us to check on your request.

1.) 2 System Support Utility logs of your system. One from the main server and one from the other server as a failover in case the main fails. This will allow us to check your Adapter details and configuration. Kindly refer to the steps below.

a- Go to https://downloadcenter.intel.com/product/91600/Intel-System-Support-Utility and download the software.

b - Open this link, for the process on how extract the logs

https://www.intel.com/content/www/us/en/support/articles/000008563/software/software-applications.ht...

2.) Kindly provide the PBA and serial number of the adapters. You may refer to the link below on where to find the PBA number. You may also provide photos of the adapters focusing on the markings (white sticker) found on the physical card for us to double check on it. The PBA is consist of 6-digit number located at the last part of the serial number.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products...

3.) Please share the link where the firmware and driver of XXV710 were downloaded. 

 

Looking forward to your reply.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

We just want to check if you still have questions regarding this thread?

If you have, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

0 Kudos
Highlighted
Beginner
87 Views

Yes we still have an issue.

 

The server is RHEL 7.7 and therefore your ssu comes out with unsupported. The servers are in a remote datacentre so I cannot provide you with the serial numbers. The driver software was downloaded from Intel - iavf-3.7.53-1.x86_64.rpm is the package.

 

0 Kudos
Highlighted
Beginner
87 Views

However altering your ssu script to accept the RH version I can get output, however the OS is not full blown RHEL - it is an appliance version - so normal Redhat packages will not install (such as ifconfig), so the output just details server info. What do you want me to do with the output?

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

Thank you for the reply. Let me gather the following details from your reply.

 

  1. Can you share the link of the driver and firmware that you used?
  2. Can you also send the output that you got for us to check if that is enough for our checking.
  3. Can you ask someone onsite to take a photo of the Ethernet cards with the markings?

 

If you have, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

I just want to make a follow up on the following information that I requested.

 

  1. Can you share the link of the driver and firmware that you used?
  2. Can you also send the output that you got for us to check if that is enough for our checking.
  3. Can you ask someone onsite to take a photo of the Ethernet cards with the markings?

 

If you have, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

0 Kudos
Highlighted
Beginner
87 Views

  1. https://downloadcenter.intel.com/download/29186/Intel-Network-Adapter-Virtual-Function-Driver-for-In... - I notice that there is now a new version.
  2. The output is too long to include in a post and won't give you anything related to the Intel cards as the commands used in the gathering of information are not available in the RHV appliance - if there was a particular command that you needed output from, I can copy it from a fully fledged RH machine and run it and supply the output, but this needs to be a targeted approach.
  3. Unfortunately not - they are in a datacentre and are live.

 

I know the above is probably unhelpful, here's what I can get out of the system:-

 

[root@htl-kvm-01 ~]# modinfo iavf

filename:      /lib/modules/3.10.0-1062.1.2.el7.x86_64/updates/drivers/net/ethernet/intel/iavf/iavf.ko

version:       3.7.53

license:       GPL

description:   Intel(R) Ethernet Adaptive Virtual Function Driver

author:        Intel Corporation, <linux.nics@intel.com>

alias:         i40evf

retpoline:     Y

rhelversion:   7.7

srcversion:    CF91EE4E4AB2130BE4FF8DF

alias:         pci:v00008086d00001889sv*sd*bc*sc*i*

alias:         pci:v00008086d000037CDsv*sd*bc*sc*i*

alias:         pci:v00008086d00001571sv*sd*bc*sc*i*

alias:         pci:v00008086d0000154Csv*sd*bc*sc*i*

depends:

vermagic:      3.10.0-1062.1.2.el7.x86_64 SMP mod_unload modversions

 

lspci output from first interface on the first card:-

 

af:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)

       Subsystem: Intel Corporation Ethernet Network Adapter XXV710-2

       Physical Slot: 4

       Flags: bus master, fast devsel, latency 0, IRQ 42, NUMA node 1

       Memory at f0000000 (64-bit, prefetchable) [size=16M]

       Memory at f2000000 (64-bit, prefetchable) [size=32K]

       [virtual] Expansion ROM at f2080000 [disabled] [size=512K]

       Capabilities: [40] Power Management version 3

       Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

       Capabilities: [70] MSI-X: Enable+ Count=129 Masked-

       Capabilities: [a0] Express Endpoint, MSI 00

       Capabilities: [100] Advanced Error Reporting

       Capabilities: [140] Device Serial Number 88-76-b8-ff-ff-fe-fd-3c

       Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

       Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

       Capabilities: [1a0] Transaction Processing Hints

       Capabilities: [1b0] Access Control Services

       Capabilities: [1d0] #19

       Kernel driver in use: i40e

       Kernel modules: i40e

 

lspci output from the first interface on the second card:-

 

b0:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)

       Subsystem: Intel Corporation Ethernet Network Adapter XXV710-2

       Physical Slot: 6

       Flags: bus master, fast devsel, latency 0, IRQ 44, NUMA node 1

       Memory at ed000000 (64-bit, prefetchable) [size=16M]

       Memory at ef000000 (64-bit, prefetchable) [size=32K]

       [virtual] Expansion ROM at ef080000 [disabled] [size=512K]

       Capabilities: [40] Power Management version 3

       Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+

       Capabilities: [70] MSI-X: Enable+ Count=129 Masked-

       Capabilities: [a0] Express Endpoint, MSI 00

       Capabilities: [100] Advanced Error Reporting

       Capabilities: [140] Device Serial Number 88-76-b8-ff-ff-fe-fd-3c

       Capabilities: [150] Alternative Routing-ID Interpretation (ARI)

       Capabilities: [160] Single Root I/O Virtualization (SR-IOV)

       Capabilities: [1a0] Transaction Processing Hints

       Capabilities: [1b0] Access Control Services

       Capabilities: [1d0] #19

       Kernel driver in use: i40e

       Kernel modules: i40e

 

Paul

 

0 Kudos
Highlighted
Beginner
87 Views

I've just realised that the info above is not complete, the physical interfaces use the i40e driver (which has a driver version 2.8.10-k), the virtual functions use iavf (i40evf is removed and aliased to iavf). We are only using virtual functions of the card.

 

The lspci for the first virtual interface:-

 

af:02.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)

       Subsystem: Intel Corporation Device 0000

       Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-

       Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

       Latency: 0

       NUMA node: 1

       Region 0: [virtual] Memory at dbfff000000 (64-bit, prefetchable) [size=64K]

       Region 3: [virtual] Memory at dbfff500000 (64-bit, prefetchable) [size=16K]

       Capabilities: [70] MSI-X: Enable+ Count=5 Masked-

               Vector table: BAR=3 offset=00000000

               PBA: BAR=3 offset=00002000

       Capabilities: [a0] Express (v2) Endpoint, MSI 00

               DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us

                       ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W

               DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-

                       RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-

                       MaxPayload 128 bytes, MaxReadReq 128 bytes

               DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-

               LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <2us, L1 <16us

                       ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+

               LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-

                       ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

               LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

               DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported

               DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled

               LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-

                        EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

       Capabilities: [100 v2] Advanced Error Reporting

               UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

               UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

               UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-

               CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

               CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-

               AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-

       Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)

               ARICap: MFVC- ACS-, Next Function: 0

               ARICtl: MFVC- ACS-, Function Group: 0

       Capabilities: [1a0 v1] Transaction Processing Hints

               Device specific mode supported

               No steering table available

       Capabilities: [1d0 v1] Access Control Services

               ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

               ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

       Kernel driver in use: vfio-pci

       Kernel modules: iavf

 

 

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

The link that you provided is just for the driver. How about the link for the firmware?

And were you able to update the firmware?

 

If you have, please let us know.

 

Best regards,

Michael L.

Intel Customer Support Technicians

A Contingent Worker at Intel

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

Good day!

 

Kindly share the download link you have used for the firmware and the exact firmware version installed.

 

Please also confirm if the latest BIOS is loaded on your system.

 

Looking forward to your response.

 

Best regards,

Crisselle C.

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Highlighted
Beginner
87 Views

The firmware will be at the version it was delivered with although the HP SSU utility may have updated it - unfortunately I cannot get to the version number from the servers iLO so I would have to down the server to possibly get to the version number. I've not been able to find a link to the firmware - only to the driver. As it is an Intel card, can you share where the firmware can be downloaded from and how to update it?

 

The HP BIOS is at the latest.

 

Paul

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

Thank you for the quick response.

 

Will you be able to identify if you are using a retail version of Intel Ethernet adapter or if it is an OEM version based on your place of purchase? The firmware that can be downloaded on our website is intended for retail version of Intel Ethernet adapter only. If you are using an OEM version, the firmware can be obtained thru your manufacturer.

 

Looking forward to your response.

 

Best regards,

Crisselle C.

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Highlighted
Beginner
87 Views

The network cards were purchased separately from the server so will be the retail version. I cannot find any place on your website to download the firmware for the xxv710 cards at all.

 

Paul

0 Kudos
Highlighted
Beginner
87 Views

Is the Non-Volatile Memory Update actually the firmware?

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

Appreciate your reply.

 

We'd like to inform you that there are also OEM Ethernet adapters that can be purchased separately and the only way for us to identify if it is a retail one is thru the markings that can be seen on the white sticker of the physical adapter. 

 

To answer your question, yes, the Non-Volatile Memory Update Utility is used to update the firmware of the adapter. 

 

The instructions on how to use the utility are also included on the download link. However, as mentioned above, this update are only intended for Intel retail Ethernet adapter.

https://downloadcenter.intel.com/download/28842/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Eth...

 

Best regards,

Crisselle C.

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Highlighted
Beginner
87 Views

I've shutdown the host so I can get more information:-

 

UEFI Driver Intel 40GbE 2.4.05

Adapter PBA J48306-003

Chip Type Intel X710

 

I'm guessing it is the PBA you need?

 

Paul

0 Kudos
Highlighted
Beginner
87 Views

Using the nvmupdate64e linux utility on the system that isn't giving us problems (due to the VM we are running blocking access to the main PF on the machine that is), I can see that the NVM Version is 6.01 so we can update the firmware and that give me something to research.

 

Paul

0 Kudos
Highlighted
Beginner
87 Views

Do you give out the release notes for the firmware versions and what they fix as there doesn't seem to be anything in the NVM Update zip?

 

Paul

0 Kudos
Highlighted
87 Views

Hello DTeam,

 

Thank you very much for the time and effort in getting the PBA.

 

After checking on it, we have confirmed that you are using a retail version of Intel Ethernet adapter.

 

I have also checked the NVM Update.zip to check if there is any information about the release, but it seems that there isn't any. With this, please give us some time to check this with our engineers. Rest assured that we will get back to you within 1-3 business days.

 

Hoping for your patience.

 

Best regards,

Crisselle C.

Intel Customer Support

A Contingent Worker at Intel

0 Kudos