Improve VMQ/RSS CPU core loadbalancing during traffic spikes

gveld · ‎08-11-2016

Some of my servers become unresponsive during traffic spikes (e.g. small DoS/DDoS attacks, which do not max out our connection.)

The servers become unresponsive because of high CPU usage, after running some tests I saw that only one core is being utilized for 100% during a <200Mbps packet flood. The server stopped replying to ICMP echo/ping.

I think the pictures explain them self.

On this machine we use VMQ, there are 7 Vport available. But according to the Intel driver only two are actually in use.

How can we allow VMQ/network stack on the host OS to use more than 1 core?

We've seen similar issues with servers with RSS load balancing, although the load balancing works better there. (spreads over multiple cores.)

Are there any tips to improve RSS load balancing during traffic spikes? We use Intel Xeon E3 systems with 4 cores + Hyper Threading and one Intel Xeon E5 system with 6 cores and Hyper Threading.

We want to keep the server working during traffic spikes and all round keep our latency to the minimum.

idata · ‎08-11-2016

Hi gijs007,

Thank you for the post. Just to double check if you refer to this document about the RSS

http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000006703.html

As noted "RSS is not supported on some adapters configured to use Virtual Machine Queues (VMQ). On these adapters VMQ will take precedence over RSS. RSS will appear disabled"

Can you also share what is the exact server system model used here? What is the onboard NIC model?

Thanks,

wb

gveld · ‎08-12-2016

I use a Supermicro SuperServer 5017C-TF, the onboard NIC model is an Intel® 82574L.

I understand that RSS is being disabled, but shouldn't VMQ still allow the host OS to use more than one core to handle network related tasks?

idata · ‎08-14-2016

Hi gijs007,

Thank you for the information. Have you also check with Supermicro on the setting as this is an onboard NIC, there could be some setting changed on the NIC when integrated on the system.

Please feel free to update me.

rgds,

wb

gveld · ‎08-16-2016

The NIC with the issue is an Intel I350-T2.

The onboad nic doesn't even support VMQ.

If I were to contact SuperMicro they would tell me they use a network chip made by Intel and that I use drivers made by Intel, so I would need to contact you guys as you wrote this software not them.

idata · ‎08-16-2016

Hi gijs007,

Thank you for the information. I will try further check for you, the reason we refer you to contact Supermicro is because they might have their customized driver that is best for the onboard NIC.

Please refer to http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000006628.html?wapkw=oem+network

rgds,

wb

gveld · ‎08-21-2016

Still I don't see why that matters, as I'm not using the onboard NIC.

I'm using the separate bought addon card: Intel I350-T2

idata · ‎08-21-2016

Hi gjs007,

Thank you for the clarification. I will check on this.

rgds

wb

idata · ‎08-23-2016

Hi gijs007,

Have you tried checking the RSS guide for reference?

http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000006703.html

Here are some reference information for performance tuning

http://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000005811.html

rgds,

wb

gveld · ‎08-26-2016

RSS is disabled as VMQ takes precedence. I'm trying to make VMQ use more than one core for the host OS.

idata · ‎08-28-2016

Hi gijs007,

I will check on this.

rgds,

wb

idata · ‎08-30-2016

Hi gijs007,

Good day. Further checking , this is an OS or system issue, you need to seek support from the OS or system vendor.

Hope the above clarified.

Thanks,

wb