I am working on a board with Dual Intel Xeon E5-2600L series processors. With our app, I am able to achieve 40Gbps processing at "X" bytes for a few minutes, with 4 LoadBalancer Cores and 16 Worker cores. However, during extended test-runs, I am seeing packet drops in large bursts (burst of a few thousand packets) even at 2X packet size over a 24hr traffic test (constant 40Gbps traffic).
I do not think it is PPS performance issue. The drops are always on the 82599 due to FIFO over-flow i.e. indicate that the Load Balancer is not fast enough to pull it sometimes (so issue is not at Workers). All the Load Balancer threads are running on Socket 0 - which is directly connected to 2 82599 devices. Theoretically the LoadBalancers should be capable of pulling 40Gbps at "X" Bytes and I also see that for a few minutes there are no drops. As I keep increasing the Packet size, the duration for the bursty drops keeps increasing, so much so that at 2X it takes about 24 hrs for the drops to happen - but I still eventually see the drops - which then again go away for next few hours and so on (I am not yet sure if there is a definite periodicity to the drops).
Inorder, to improve the LoadBalancer thread performance, I do not have those threads Hyper-threaded with the Worker threads - i.e. the corresponding HyperThread is not allocated (Initially I had the Workers hyper-thread with LoadBalancer and that was impacting performance of the LoadBalancer - resulting in drops on 82599). I believe, the non-allocated threads can be used by the Linux Kernel. I was therefore wondering if this sudden drop indicates that some Linux Kernel processing is happening on the free cores and holding off the LoadBalancer due to hyper-thereading. To test this, I wanted to fix the Linux cores, to the other 4 cores that I have available (and ensure that there is no hyper-threading with the LoadBalancer threads). Is there a way I can force the Linux Kernel to exclude certain cores?
Any other suggestions as to what maybe causing these drops during an extended traffic run (at line-rate) are welcome. (I do plan to test with the sample LoadBalancer app soon).
Can you kindly let us know your update on this please.
Forcing the Linux kernel to exclude certain core - you must be familiar with ISOLCPU - that isolates the core from Kernel. https://groups.google.com/forum/# !topic/linux.kernel/Txr5lt--Vv8 https://groups.google.com/forum/# !topic/linux.kernel/Txr5lt--Vv8
Please let us know