Loss of network in guest VM's on Nutanix cluster on nodes with Intel X520 nics

[stouttech]

We're running a 5-node Nutanix cluster with (3 nodes with Intel NIC's, 2 nodes with Mellanox NIC's). Since the Windows updates in May 2022, we've had an odd issue with our 3 nodes that have the Intel NIC's where after a restart of the host, any guest VM running Server 2019 that gets migrated back to the host loses network access. If we restart the host and let it sit for about 4-5 hours, we can migrate Sever 2019 machines to it without issue. Other OS's (2012, 2016, linux) work just fine and we don't have to wait. The only thing that we've found is that if we manually revert the Hyper-V network adapter on the guest VM to teh version prior to the May 22 update, it works. We've reached out to Nutanix and Microsoft and haven't gotten anywhere so just checking here to see if anyone has experienced anything similar. 

Thanks. 

[B_Y]

Hi UW Stout,

Thank you for contacting Intel Ethernet Community.

We have not hear anyone else experiencing the similar issue at this moment.

It may seems when a big Windows update is performed, the VLAN (Virtual Local Area Network) configuration is lost and must be re-added. May I know if the Virtual Machine Queues (VMQ) is enabled in host NIC?

Attached an article that may helpful in this scenario:

<https://learn.microsoft.com/en-us/troubleshoot/windows-server/virtualization/network-connectivity-lost-on-hyper-v-vms>

Best Regards,

BY_Intel 

[B_Y]

Hi UW Stout,

We would like to follow up on your case regarding Loss of network in guest VMs.

We have not hear anyone else experiencing the similar issue at this moment.  In our last interaction, we inquired whether the Virtual Machine Queues (VMQ) is enabled in your host NIC.

We would like to know if the provided article was able to help in your situation. In case you did, we would highly appreciate if you would let us know the outcome of these. We will be waiting for your response.

Best Regards,

BY_Intel 

[B_Y]

Hi Adam,

I wish to gather some information below for further checking:

1) Are all these VMs in production?

2) May I know which version of Windows Update (KBxxxx) that seemly cause the issue after the installation on the Intel NIC?

3) May I know the exact models for each Intel NICs? 

Please follow these steps:

Download Intel® System Support Utility for Windows* software. When the download is complete, launch SSU.exe.

1. Scan: Check the box Everything.
2. Click Scan.
3. Review: When finished scanning, click Next.
4. Click Save.
5. At the network adapts, it should show the model of the NIC

4) I wish to know more of your Nutanix cluster, is Intel NIC is the host NIC?

Best Regards,

BY_Intel

[B_Y]

Hi Adam,

Some questions for us to pursue this:

1) What are the exact steps to reproduce the issue?

-“… after a restart of the host, any guest VM running Server 2019 that gets migrated back to the host loses network access.”

-Does this mean the issue occurs on a Windows Server 2019 VM on system A after it’s been live migrated from system A to system B, then back to System A? I’m not certain that I’m following this correctly.

2) Is the SSU log from the host system or the VM (I assume the host but want to make sure)? Can you send the SSU log from the VM?

3) What version of Intel virtual driver are you running on the VM (SSU log from VM would tell us)?

4) Which adapter(s) have the issue (I see both X520 and X722 in the SSU log)?

5) Have you tried updating the Intel ethernet drivers on the host system (they’re both 2-3 years old)?

-The X722 Intel driver in the SSU log is version 1.13.104.0 from 08/31/2020. Release 27.8 has version 1.16.202.6.

-The X520 Intel driver in the SSU log is version 4.1.197.0 from 11/27/2019. Release 27.8 has version 4.1.239.0.

Best Regards,

BY_Intel

[B_Y]

Hi Adam,

Hope you're doing well.

Since we have not seen an update for several days, if you need any additional information, please submit a new question as this thread will no longer be monitored.

Thank you for choosing Intel. 

Best regards,

BY_Intel