UDP Dropped Packet on Optiplex 7060 ( Intel Xeon E3 1200/1500 )

MPron · ‎02-22-2019

Hello,

We are experiencing Network troubles on the Dell OptiPlex 7060 Computer with Windows 10 Version 1803 and 1809.

This affect greatly the UDP performance for one of our application.

To Demonstrate this we used Iperf3 on another computer with a direct network link.

Here's the output on the OptiPlex Computer :

-----------------------------------------------------------

Server listening on 1001

-----------------------------------------------------------

Accepted connection from 192.168.16.10, port 62268

[ 6] local 192.168.16.1 port 1001 connected to 192.168.16.10 port 60459

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 6] 0.00-1.00 sec 7.57 MBytes 63.5 Mbits/sec 0.215 ms 159/1128 (14%)

[ 6] 1.00-2.00 sec 12.0 MBytes 100 Mbits/sec 0.221 ms 29/1561 (1.9%)

[ 6] 2.00-3.00 sec 11.8 MBytes 99.4 Mbits/sec 0.250 ms 0/1516 (0%)

[ 6] 3.00-4.00 sec 10.4 MBytes 87.2 Mbits/sec 0.279 ms 152/1482 (10%)

[ 6] 4.00-5.00 sec 11.6 MBytes 97.7 Mbits/sec 0.267 ms 73/1564 (4.7%)

[ 6] 5.00-6.00 sec 11.8 MBytes 99.2 Mbits/sec 0.244 ms 0/1514 (0%)

[ 6] 6.00-7.00 sec 12.0 MBytes 101 Mbits/sec 0.242 ms 1/1541 (0.065%)

[ 6] 7.00-8.00 sec 10.7 MBytes 90.0 Mbits/sec 0.266 ms 136/1509 (9%)

[ 6] 8.00-9.00 sec 12.1 MBytes 101 Mbits/sec 0.247 ms 1/1544 (0.065%)

[ 6] 9.00-10.00 sec 11.8 MBytes 99.0 Mbits/sec 0.252 ms 18/1529 (1.2%)

[ 6] 10.00-10.20 sec 2.22 MBytes 95.2 Mbits/sec 0.249 ms 0/284 (0%)

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 6] 0.00-10.20 sec 0.00 Bytes 0.00 bits/sec 0.249 ms 569/15172 (3.8%)

-----------------------------------------------------------

In some cases we even had a Lost packet ratio that goes well beyond 20 %.

We suspect that this behavior is linked with an Upgrade from some drivers ( ?? ) :

- If we install a fresh 1809 version without being connected to the internet : the tests with iperf3 are pretty good : 0% of Lost Packets

- if we do the same , but we instead connect the computer to the internet, we experience a high ratio of lost Packets. We tried then to install drivers from intel and Dell to go back to a working state : this had no effect on the network performance.

Has someone any idea of :

- the source of the problem ? we think it's around the intel 300 chipset ( we have no real clue )

- how to solve the problem after being connected to the internet ? ( rolling back to version that worked )

PS :

We made the same test in Safe Mode with Networking.

And this is quite surprising...

Here's the output from iperf3 :

-----------------------------------------------------------

Server listening on 1001

-----------------------------------------------------------

Accepted connection from 192.168.16.10, port 60193

[ 6] local 192.168.16.1 port 1001 connected to 192.168.16.10 port 64307

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 6] 0.00-1.01 sec 7.25 MBytes 60.1 Mbits/sec 0.118 ms 0/928 (0%)

[ 6] 1.01-2.01 sec 12.0 MBytes 100 Mbits/sec 0.141 ms 0/1531 (0%)

[ 6] 2.01-3.00 sec 12.1 MBytes 103 Mbits/sec 0.232 ms 0/1548 (0%)

[ 6] 3.00-4.01 sec 11.9 MBytes 98.7 Mbits/sec 0.222 ms 0/1524 (0%)

[ 6] 4.01-5.01 sec 11.7 MBytes 98.5 Mbits/sec 0.109 ms 0/1502 (0%)

[ 6] 5.01-6.00 sec 12.0 MBytes 102 Mbits/sec 0.258 ms 0/1538 (0%)

[ 6] 6.00-7.00 sec 12.0 MBytes 100 Mbits/sec 0.246 ms 0/1530 (0%)

[ 6] 7.00-8.00 sec 11.6 MBytes 97.6 Mbits/sec 0.200 ms 0/1490 (0%)

[ 6] 8.00-9.01 sec 12.2 MBytes 101 Mbits/sec 0.209 ms 0/1562 (0%)

[ 6] 9.01-10.00 sec 12.0 MBytes 102 Mbits/sec 0.212 ms 0/1535 (0%)

[ 6] 10.00-10.37 sec 3.57 MBytes 80.2 Mbits/sec 0.201 ms 0/457 (0%)

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams

[ 6] 0.00-10.37 sec 0.00 Bytes 0.00 bits/sec 0.201 ms 0/15145 (0%)

-----------------------------------------------------------

Server listening on 1001

-----------------------------------------------------------

Crisselle_Intel · ‎02-25-2019

Hello MPron, Thank you for posting in Intel Ethernet Communities. Please provide the System Support Utility log of your system. This will allow us to check your Adapter details. Kindly refer to the steps below. 1- Download from https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows- 2- Open SSU.exe 3- Mark the box "Everything" and then click "Scan". 4- When finished scanning, click "Next". 5- Click on "Save" and attach the file to a post. If you have any other questions, please let us know. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

MPron · ‎02-25-2019

Hello Crisselle,

I executed the binary on the computer.

You'll find the txt in attachment

Best Regards,

Crisselle_Intel · ‎02-26-2019

Hello MPron, Thank you for the response. We will check the provided logs and we'll get back to you as soon as possible. If you have any other questions, please let us know. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

Crisselle_Intel · ‎02-27-2019

Hello MPron, Thank you for waiting for our update. May we know if you already tried the latest driver from SW 23.5.2? https://downloadcenter.intel.com/download/22283/Intel-Ethernet-Adapter-Complete-Driver-Pack?product=82185 If you have any other questions, please do not hesitate to ask. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

MPron · ‎02-27-2019

Hello Crisselle,

We applied this modification without any effect on the ratio of dropped packet.

Best Regards ,

Crisselle_Intel · ‎02-28-2019

Hello MPron, Thank you for the response. While we are checking on this issue, we just want to clarify if you used Intel network driver after Windows installation? Or you already test to see if there are lost packets after installing Windows 10 1809(without installing Intel Network driver)? If you have any other questions, please do not hesitate to ask. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

MPron · ‎02-28-2019

Hello Crisselle,

So here are the tests we made :

fresh install of 1809 without any connection to the network : No loss of UDP packets
fresh install of 1809 without any connection + install of intel ( or dell ) drivers and without any connection to the network : No loss of UDP packets
fresh install of 1809 connected to the network : loss of UDP packets
fresh install of 1809 without any connection to the network then blocking of the driver update. Computer connected to the internet : No loss of UDP packets
fresh install of 1809 without any connection to the network , install of manufacturer driver then blocking of the driver update. Computer connected to the internet : No loss of UDP packets
Upgrade of non working 1803 correctly to 1809 : loss of UDP packets . In this situation we didn't find a way to roll back to a working state
1803 with Manufacturer Driver : loss of UDP packets
1809 with Manufacturer Driver : loss of UDP packets

We'd really like to find a way to go back to a working state from a 1803 or 1809 without installing a fresh win10 1809 install.

we have the same type of problems with some Lenovo P330

Best regards,

Crisselle_Intel · ‎03-01-2019

Hello MPron, Thank you for the information. We will check on this and we will get back to you as soon as possible. If you have other questions, please do not hesitate to ask. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

Crisselle_Intel · ‎03-04-2019

Hello MPron, Thank you for the patience. May we know if you tried using other brand of Ethernet card on your system to further isolate the issue? We just want to clarify that if you install a fresh Win10 1809 without updating the NIC driver then connect to network and internet there is no loss of UDP packets? If you have other questions, please do not hesitate to ask. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

MPron · ‎03-04-2019

Hello Crisselle,

We made some other tests using other other network adapter and add the same problem ( we even use a Usb3 to Ethernet adapter [Startech] that work fine on other system).

We don't really think that the problem is link to the network adapters.

To clarify your last point :

install of a fresh Win10 1809 without any connection to internet
Block the upgrade of Drivers in windows
Connect to the network (and internet) : we have no loss of UDP packets and our application is working as expected

Regards,

Crisselle_Intel · ‎03-05-2019

Hello Mpron, Thank you for the reply. We will check again the issue and well update you as soon as possible. Please let me know if you have other questions. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

MPron · ‎03-05-2019

Hello Crisselle ,

We have additional information concerning this trouble :

On a working system ( 1809 ), if we update the Standard SATA AHCU Controler from the device Manager, we obtain the following version :

Intel(R) 300 Series Chipset Family SATA AHCI Controller in version 16.8.0.1000

We then test it with our application and we have the UDP problem.

If we rollback the driver to the version distributed by microsoft, the problem disappears ( No more UDP problems )

If this problem known ?

Best Regards,

Crisselle_Intel · ‎03-06-2019

Hello Mpron, Thank you for the update. May we know if you tried contacting Dell for this issue since using 3rd party Ethernet adapter also experienced the same problem? Please let me know if you have other questions. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

MPron · ‎03-06-2019

Hello Criselle,

As we have the same problem with the Lenovo P330 ( with a much bigger install base ) , we are in contact with Lenovo to understand the problem.

I like to add that the AHCI controller is indeed not directly link with the network card. We made additional tests, and confirm that this intel driver is faulty regarding the UDP performance.

Can someone from your organisation confirm that the Intel(R) 300 Series Chipset Family SATA AHCI Controller in version 16.8.0.1000 is not working as expected ?

Best Regards,

Crisselle_Intel · ‎03-07-2019

Hello Mpron, Thank you for the reply. We will further check on this and we will get back to you as soon as possible. Is it possible to update us of the result after contacting Lenovo? If you have other questions, please let me know Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

Crisselle_Intel · ‎03-08-2019

Hello Mpron, Thank you for waiting. Is it possible if you can share additional information that the confirmed part that Intel driver is at fault regarding the UDP performance? Can you provide SSU logs from your Lenovo P330? If you have other questions, please let me know Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

MPron · ‎03-08-2019

Hello Crisselle,

Here's SSU Logs we produce on our Lenovo P330.

This is made in the problematic situation ( with the Intel(R) 300 Series Chipset Family SATA AHCI Controller in version 16.8.0.1000 )

Mike_Intel · ‎03-11-2019

Hello Mpron, Thank you for the reply. We will check again the issue and well update you as soon as possible. Please let me know if you have other questions. Best regards, Michael L. Intel Customer Support Under Contract to Intel Corporation

Crisselle_Intel · ‎03-12-2019

Hello Mpron, While we are checking the logs, we just want to share this information. We have UDP issue stated in readme file but is applicable to ixgbe driver, sharing with you as reference. UDP Stress Test Dropped Packet Issue Under small packet UDP stress with the ixgbe driver, the system may drop UDP packets due to socket buffers being full. Setting the driver Intel Ethernet Flow Control variables to the minimum may resolve the issue. https://downloadmirror.intel.com/14687/eng/readme.txt Please let me know if you have other questions. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation

Crisselle_Intel · ‎03-13-2019

Hello Mpron, Since the issue was caused by the Intel RST driver 16.8.0.1000, we suggest to continue working with OEM (Lenovo and Dell) for the issue and also post your observation in the RST forum for assistance. https://forums.intel.com/s/topic/0TO0P00000018NXWAY/rapid-storage-technology?language=en_US Please let us know if there's other assistance needed from the Ethernet forum. Best regards, Crisselle C Intel Customer Support Under Contract to Intel Corporation