Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4809 Discussions

Bad Windows 10 Pro networking performance using an X550-T2 converged network adapter

ma-hnln
Beginner
4,301 Views

Hi,

I am working with a X550-T2 converged network adapter and would like to receive 10 Gb/s of UDP packets on some amount (e.g. 4) of target ports from the same source IP address. To check if my hardware and driver setup generally works, I am using iperf to generate and receive as much traffic as possible. Everything works perfectly fine (stability and performance wise) when running the receiving part of iperf using Linux, but not when using Windows 10 Pro (both on the same underlying hardware).

As iperf is not optimised for Windows, I additionally wrote a WinSock Registered IO receiver and came to the same conclusion than the iperf test: under Windows 10 Pro, I get very bad networking performance.

From there I simplified the test setup and tried single socket performance and only got to around 180k packets (@ size of 1k) per seconds without major drops. Beyond this number, drops get so overwhelming that is basically impossible to do anything with the incoming data. I also tried simple, blocking IO and came to the same conclusion: performance and stability is very bad under Windows 10 Pro.

To ensure that my code was not the cause of the observed behaviour on Win 10, I downloaded test setups from this series of blog posts . The results were equally bad. Additionally, the posts show that the expected amounts of packets per second, should be indeed much larger (i.e. >= 450k). Most interestingly, these numbers are from 2012, therefore I would expect to beat them with today's hardware and not be slower by a factor of almost 3.

I have configured the network adapter appropriately, e.g.: turning all UDP offload features on and off, maxing out the receive buffer, maxing out interrupt moderation, activating DMA coalescing, ...). Also, the receiving machine does not use SMT/HyperThreading. There is no active firewall and sender and receiver are connected back to back via a link that has proven to be capable of 10 Gb/s (via Linux).

Why is this test under perfect lab conditions working out well on Linux and basically not at all on Win10 Pro?

 

0 Kudos
31 Replies
Mike_Intel
Moderator
3,177 Views

Hello ma-hnln,

 

Thank you for posting in Intel Ethernet Communities. 

 

Before we proceed, let me check. Are you designing a system with emended X550-T2?

 

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

 

Best regards,

Michael L.

Intel® Customer Support

 

0 Kudos
ma-hnln
Beginner
3,172 Views

Hi Michael,

thanks for your response.

I am working with a typical desktop application and my test setup uses two typical desktop machines too. So no, there is nothing embedded going on.

Spreading the load over 10 sockets (4 RSS queues) yields somewhat stable 300k packets per second. However, there is still potential for massive drops (from which the system does not recover for a long time) and I now see some kind of flow control kicking in - but I have disabled flow control on both the sending and the receiving adapter. 

Sometimes, I can observe very large packet per second counts (around 900k, with the same setup as above), but at this point the drop rate is continuously high and renders the communication useless.

The more I am interacting with this problem, the more it seems that Windows itself is the Problem. But maybe its something about the driver (which I updated just now to the latest version without any effect).

0 Kudos
ma-hnln
Beginner
3,169 Views

I'm curious, which data rates do Intel engineers achieve under Windows 10 Pro when testing their drivers for performance? The packet rates I receive in a somewhat stable way might already be the maximum one can expect under Windows.

0 Kudos
ma-hnln
Beginner
3,168 Views

Sorry, I meant to ask "which packet rates" the engineers achieve.

0 Kudos
Mike_Intel
Moderator
3,144 Views

Hello ma-hnln,


Thank you for posting in Intel Ethernet Communities. 


By the way, can you tell us why do you need this information? What is your end goal for this test?


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
ma-hnln
Beginner
3,139 Views

Why do I need this? Because I am forced to use a Windows 10 (not server) setup to receive large amounts of UPD traffic (up to 10 Gb/s, latency is not a major concern).

Further, I need this process to be as stable as it gets. Yes, UDP is not designed to guaranty delivery but as I usually work with simple, local networks and with a sender that would never be able to fulfil the concept of TCP, I am stuck with UDP. This also means that the receiver (most likely) is the single point of failure - and I see that it fails terribly.

Additionally, I can't always rely on "jumbo" packets to decrease the amount of packets per second. This leads me to where I am now: testing performance and stability for large amounts of 1k packets per second and observing that it does not work as one would expect when using a 10 Gb/s Intel network adapter.

One last point: as long as I don't see a somewhat stable and large amount of small packets per second I always have to assume that lower counts of large packets are also not handled as well as they should be handled, leading to sub par performance. 

0 Kudos
Mike_Intel
Moderator
3,128 Views

Hello ma-hnln,


I understand, let me gather more details for me to better understand your set up or configuration.


  1. Are you using onboard X550-T2 or a PCIe card?
  2. If you are using a PCIe card, can you send photos of your network card on both sides focusing on the markings?
  3. Can you share the link of your latest driver?
  4. Please download and generate the SSU of your system and send it to me via direct message.

Download from https://downloadcenter.intel.com/download/25293/Intel-System-Support-Utility-for-Windows-


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
ma-hnln
Beginner
3,124 Views

Tanks for your response - I really appreciate you taking the time for shedding some light on this issue.

Unfortunately, I'll need a day or two to get back to this problem again and provide the information you have requested.

0 Kudos
Mike_Intel
Moderator
3,112 Views

Hello ma-hnln,


That's okay, I will give you the weekend to provide an update on this one.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
Mike_Intel
Moderator
3,028 Views

Hello ma-hnln,


We hope you had a great weekend. We just want to make a follow up on the information that we requested for us to further check this issue.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
ma-hnln
Beginner
3,024 Views

Hi Michael,

I will send you the requested data within a day. Sorry for the delay.

0 Kudos
ma-hnln
Beginner
3,005 Views

I have send you everything via a private message.

 

0 Kudos
Mike_Intel
Moderator
2,997 Views

Hello ma-hnln,


Thank you for sending the information that we requested. Upon initial checking, you are already using the latest driver, however can you try the following steps for the efficient installation of the driver.


Try to complete uninstall the driver and software and install the latest version of the driver again.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
ma-hnln
Beginner
2,979 Views

Thanks for your swift response, Michael. I'll come back to you once I'm done with it.

Additionally it seems that the issue described in https://community.intel.com/t5/Ethernet-Products/Why-I-have-loss-during-capture-data/td-p/1217208/page/4 is similar to mine.

When reading the thread the user states that RSS indeed spreads the load across cores but does not reduce the data loss overall (this is exactly what I observe too). Further, I'm not convinced that using more than a single Ethernet port of the network card is the actual problem in this thread - it might just as well be the higher throughput resulting from more than one Ethernet port being used. Maybe its just Windows failing after all.

0 Kudos
Mike_Intel
Moderator
2,968 Views

Hello ma-hnln,


Thank you for the update. I will wait for the your feedback once after trying to reinstall the driver. I will check this link and also if there are other documentation done on this thread for me to also share it you.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
ma-hnln
Beginner
2,915 Views

Hi Michael,

thanks for waiting. I have completely uninstalled and then reinstalled all NIC related drivers on the receiver system.

It seems that a test receiving via 10 sockets (i.e. spreading the load) is now capable of stably handling around 430k packets per second. However, sometimes, such a setup will cause exceedingly high drop rates and sometimes not. This might be related to how the system associates packet streams with RSS queues and how these RSS queues are associated with CPU cores. As everything is unchanged between individual test runs (i.e. sender and receiver IP+port always the same) I don't get why this behaviour changes so drastically.

Further, single socket performance (the simple benchmark) is still where it was before (around 180k packets per second). As I have initially stated: I would assume single socket data transfer rates to be around 350k to 450k packets per second (under perfect lab conditions) using an API like Registered IO (as it was specifically build for this use case). It would be nice if you could tell me what packet per second numbers your engineers would expect when using such a single socket Windows based setup - maybe I'm just expecting too much (which I highly doubt).

Finally, there still seems to be some kind of flow control kicking in (despite it being disabled on sender and receiver side). This hinders the test setup from scaling to 9.5 Gb/s but as stability is bad far below this value, this is not of a big concern.

0 Kudos
Mike_Intel
Moderator
2,902 Views

Hello ma-hnln,


Thank you for the update and I am glad that we are getting some improvements after the reinstallation of the driver.

However, before we proceed. I need to further check clarify the next help needed and is it still related to the original issue that we encounter? Please further explain.


If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
ma-hnln
Beginner
2,884 Views

Hi Michael,

as I have stated in my previous reply, I am still unable to improve single socket performance above the original threshold of 180k packets per second. Additionally, in all cases (single or multi socket), stability of the data transfer is very disappointing, considering that everything is tested under perfect lab conditions.

What I want is to maximise stability and performance of UDP traffic received by a Windows 10 system using an Intel NIC. It is possible to somewhat stably receive 9.5 Gb/s when using jumbo Ethernet frames but even with this setup (which is not even possible to achieve in all real world scenarios) data transfer can become very unstable - to the point where the drop rate is constantly high and makes the actually received data useless. To get to the bottom of why this is the case, I am now working with smaller packet sizes as these show this bad behaviour more consistently. Furthermore, it seems that testing for the amount of packets per second is a benchmark that is widely performed and I therefore can get some reference numbers of expected packet rates online.

To sum it up:

  • I get 180k packets, single socket receive
  • I want to see 350k to 450k packets, single socket receive

This will then give me the confidence I need to state that every thing works as expected and with maximum stability and performance.

0 Kudos
Mike_Intel
Moderator
2,650 Views

Hello ma-hnln,


Thank you for the needed clarification. We will further check this issue on how we can improve the performance.

Please give us 3 to 4 working days to provide an update.


Thank you.


Best regards,

Michael L.

Intel® Customer Support


0 Kudos
Reply