Random slowness and network performance issues with Intel® X550-AT2 on Asrock motherboard

[ForumResearcher]

We have a Windows Server 2019 Hyper-V host with a Windows 10 VM guest with SQL installed.

The guest has slow file copy operations and SQL select queries.

The host has Intel® X550-AT2 (2x 10G) ethernet LAN ports. One port is shared with the VM using Virtual Switch and another port is dedicated with host server.

Effectively, two ports are used by host with different subnet range. The motherboard is an Asrock ROME2D16-2T (Server Rack).   
(Detailed hardware and OS Specifications are at the end of this post)

We conducted network speed tests using iPerf, and the results indicate that outgoing transfer speeds are effectively zero in the following scenarios:

1. From the VM to outside the VM
2. From the Host to outside the Host

This behavior is consistent across both network adapters on the host machine.

However, there is no issue when:

1. Copying data between drives within the VM

2. Copying data from other PCs on the network to the VM or Host (Incoming traffic)

Event Logs & IntelDCB Warning

The Event Viewer, has frequent Application Event ID 791 logged for IntelDCB, with the message:

"Application feature on a device has changed to non-operational." (Fiber Channel over Ethernet).

We referred to the Intel Ethernet controller datasheet that notes Intel Data Center Bridging (IntelDCB) is responsible for ensuring that network packets are transmitted reliably and without loss.

However, we're uncertain about the exact corrective steps.

Online Research & Attempted Fixes

Our research suggests the issue could be related to:

• Virtual switch misconfiguration
• Antivirus or firewall interference
• Corrupted NIC drivers
• Network adapter Offload settings

Virtual Machine Queue (VMQ) settings: As per one of the forum post, it refers to VMQ solving the issue. We disabled and re-enabled VMQ, but the issue persists.  - https://community.spiceworks.com/t/file-copy-from-host-vm-or-vm-vm-slow/482327/5

Additionally, CPU and memory usage on both the host and VM are within acceptable limits.

We are looking to understand:

What could be the root cause of zero outgoing packet transfers in this setup?

What troubleshooting or configuration changes might resolve it?

Troubleshooting Steps Tried:

• Connected one network port dedicated to VM
• Interchanged the adapters with VM
• Changed network cables, ports in network switch etc.
• Verified VMQ settings
• Tested with different antivirus/firewall settings
• Checked with latest NIC drivers
• Reset & configuring the virtual switch
• Enabled Receive Segment Coalescing (RSC) and later disabled

iPerf Results Summary

Test 1: Host → VM (Outgoing from host to VM) Connecting to host **bleep**, port xx
[ 4] local xxxx port **bleep** connected to **bleep** port **bleep**
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 693 MBytes 582 Mbits/sec sender
[ 4] 0.00-10.00 sec 693 MBytes 582 Mbits/sec receiver

Test 2: VM → Host (Outgoing from VM to host) Connecting to host **bleep**, port xx
[ 5] local xxxx port **bleep** connected to **bleep** port **bleep**
[ 5] 0.00-10.01 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-10.01 sec 3.90 GBytes 3.35 Gbits/sec receiver

Hardware Specifications

Host OS: Windows Server 2019

VM OS: Windows 10 with SQL Server Standard 2017

Antivirus Details: Sentinelone Singularity Control

Motherboard: ASRock ROME2D16-2T (Rack)

Processor: AMD EPYC 7373X – 16 Cores / 32 Threads, 3.05/3.80GHz, 768MB L3 Cache

Ethernet: Intel® X550-AT2 – 2× 10GbE RJ45 Ports

NICs: 2 physical network adapters

RAID Controller: LSI MegaRAID 9271-4i SGL SATA+SAS (LSI00328)

Disk Drives: WD Blue SN5000 NVMe SSD – 500GB, up to 5000 MB/s

Samsung PM893 Enterprise SATA SSD – 480GB, up to 550 MB/s

WD Red SA500 NAS SATA SSD – 2TB, up to 560 MB/s

I guess we could eliminate the network switch as a suspect based on today’s testing. Because even when we connect the affected host ( i.e host of this VM) to another host through a direct connection, without any network switches in between, we are still facing this issues. As far as the network switch is concerned, the random packet loss issue hasn’t occurred for any other devices on the same switch, either as a source or destination.

We are also checking the below community posts :

https://community.intel.com/t5/Ethernet-Products/1-out-of-2-10Gbe-X550-port-not-usable-after-firmware-upgrade/td-p/1269602/highlight/true/page/2
https://community.intel.com/t5/Ethernet-Products/Bad-Windows-10-Pro-networking-performance-using-an-X550-T2/td-p/1255236

We would appreciate any suggestions or insights from the community regarding potential causes or resolution steps. Thanks in advance.

[pujeeth]

Hello ForumResearcher,

Greetings!!

Thank you for contacting Intel.

In order to proceed further with our troubleshooting, we kindly request you to provide the following information:

1) Please let us know if the network adapter came pre-installed with the system, was purchased separately, or is an onboard NIC.

2) Kindly share the SSU (System Support Utility) logs to help us better understand the system configuration.

https://www.intel.com/content/www/us/en/support/articles/000008563/ethernet-products.html

Please feel free to reply to this email. We're here to assist you every step of the way.

Best regards,

Pujeeth

Intel Customer Support Technician

[ForumResearcher]

Hello Pujeeth, thanks for the response.
1) The Intel NIC and network adapter came pre-built with the system.
2) Regarding the SSU logs, the server is presently down due to some other reasons.  Once its up we shall run the SSU tests and send the logs. 

[Sreelakshmi1]

Hi ForumResearcher,

Greetings!!

Thank you for confirming the details. Since the Ethernet controller came with the Asrock motherboard, we recommend checking with Asrock, OEM(Original Equipment Manufacturer) regarding the random slowness and network performance issues. Using software and firmware versions that have not been validated to work together may affect performance, feature availability, and system stability.

Unfortunately we are unable to extend support for this product, as it falls outside of our support boundaries.

Thank you for understanding.

Best Regards,

Sreelakshmi

Intel Customer Support Technician

[ForumResearcher]

Hello Sreelakshmi,  thanks for the response.  Just seeking one clarification:
"Using software and firmware versions that have not been validated to work together may affect performance, feature availability, and system stability."

As per this,  is there any document from Intel which mentions tested or supported processors, for our reference?  We checked the datasheet pertaining to the Ethernet adapter but were not able to locate this specifically, if yes.

[Sreelakshmi1]

Hi ForumResearcher,

Greetings!

Thank you for your response. We would like to inform you that there is no official documentation regarding tested or supported processors for the Intel Ethernet adapter. For more information, we recommend contacting the system vendor regarding the requested information.

We will proceed to close the request.

Thank you understanding.

Best Regards,

Sreelakshmi

Intel Customer Support Technician

[ForumResearcher]

@Sreelakshmi1 @pujeeth   

 Hello Everyone,  Just wanted to come back and post saying the issue is solved now.  We reset the network driver.