X550 und Windows Server 2016 and 2019 caused a PCIE error when using a LBFO teaming when using Intel's driver 24.1 or newer

[MKrac2]

We have multiple Fujitsu Primergy RX2540 M4 systems with two additional 2 port X550 NICs present. We are using Windows Server 2016 and Windows Server 2019 and on both operating systems we got an PCIE error while we creating a LBFO teaming and adding a Hyper-V switch. We are trying to use released drivers from Fujitsu (Intel 24.1) and newest from Intel (Intel24.3). With both versions we saw same behavior. The problem is gone if we are using Intel 23.1 in Windows Server 2016 or we use built in driver from Windows Server 2019.

The issue does not occur on every machine. So we analyzed the differences between the systems. We found out, that all systems with NVM 1.0.0 on X550 cards will NOT raise the issue. If NVM 1.143.0 or newer (2.0.0) is installed than we saw the issue coming up.

We also tried to bring the system alive using Intel's driver 23.1 within Windows Server 2016 and built-in driver within Windows Server 2019.

That worked and we was able to build the LBFO teaming and adding Hyper-V switch to it.

Finally after the server was operational we tried to update to newer Intel driver 24.x and after applying that driver on a system where the NVM was on 1.143.0 or later it raised the PCIe error immediately and freeze the whole system.

Does anyone saw same behavior and has a solution for it?

Thanks,

Matthias

[Caguicla_Intel]

Hello Matthias,

Thank you for posting in Intel Ethernet Communities. 

Please provide the following information for us to check on your query.

1.) System Support Utility log of your system. This will allow us to check your Adapter details and configuration. Kindly refer to the steps below.

a- Go to https://downloadcenter.intel.com/product/91600/Intel-System-Support-Utility and download the software.

b- Open SSU.exe

c- Mark the box "Everything" and then click "Scan".

d- When finished scanning, click "Next".

e- Click on "Save" and attach the file to a post.

2.) Please share the exact steps done to create LBFO teaming and adding a Hyper-V switch.

Looking forward to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Matthias,

How are you doing today?

Please provide the requested information for us to proceed in checking you request.

Awaiting to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[MKrac2]

Hello Crisselle,

to answer your questions:

1.) System Support Utility log of your system. This will allow us to check your Adapter details and configuration. Kindly refer to the steps below.

-> Was uploaded to case.

2.) Please share the exact steps done to create LBFO teaming and adding a Hyper-V switch.

-> This story needs more line to explain it exactly.

We use Microsoft System Center Virtual Machine Manager (VMM) 2016 and 2019 for the bare metal deployment (BMD). For using it, it is necessary to configure multiple objects within VMM.

Following teaming related objects was configured:

• 2 Uplink Port Profile - Settings:
  • Load balancing algorithm: Dynamic
  • Teaming mode: Switch Independent
• 2 Logical Switches, one to be used for host/management and a second for virtual machines - Settings:
  • Uplink Mode: Team
  • Minimum Bandwidth mode: Weight
  • SR-IOV: Off
  • Extensions - Microsoft Windows Filtering Platform
• VMM Library contains driver, virtual hardisk (vhdx - was created by using a vanilla image of Windows Server and added updates using DISM)

To do the BMD we also defined a VMM physical computer profile and created a boot.wim for PXE boot.

We made different tests:

1. BMD with actual Intel drivers (we tried different versions released by Intel and Fujitsu 24.1, 24.2 and 24.3). After successful copy of operating system (OS) vhdx file to local disk and making it bootable and injecting drivers the WinPE phase ends. Now the OS will start the first time and do a unattended installation. At the end the VMM agent will be deployed to it and the VMM agent will contact the VMM server to get the next steps. At this moment the provided driver is in place and works also for the network devices. As next the VMM agent will build up the LBFO team and during that it caused a PCI hardware error when a NVM 1.143.0 or newer was active on X550 network cards. We did not saw that error with NVM 1.0.0 on X550.
2. BMD with older Intel drivers (released by Fujitsu 23.1) and the deployment was successful (most of the time, only on some servers we saw the PCI hardware error when these also had newer NVM version like 1.143.0 or 2.0.0). We updated the NVM for X550 from 1.0.0 to 2.0.0 on successfully deployed servers with Intel driver 23.1, that worked. Now after that update we tried to install an actual Intel driver (we tried different versions released by Intel and Fujitsu 24.1, 24.2 and 24.3) and every time the PCI hardware error was raised.

Best regards,

Matthias

[Caguicla_Intel]

Hello Matthias,

Thank you for providing the requested information.

While we re checking on this, can you share the exact PCI error message as well as how many systems are affected by this issue. This would help us in further investigating the issue.

Looking forward to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Matthias,

How are you doing today?

We hope you don't mind sharing the additional information that we requested that will help us in further checking this request.

Looking forward to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Matthias,

Please be informed that we will close this request temporarily since we haven't received any response from our previous follow up. You may reopen this request for the next few days and provide the additional information we requested for us to further check the issue. You may also post a new question in case you have any other concern or assistance needed in the future.

May you have a great day ahead!

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel