X710-DA4 not respecting Base Processor

RTill · ‎09-19-2017

Have a dual socket HP Z840 with NIC attached to CPU1 runing windows server 2016.

Have a X710-DA4 with 22.6 driver installed.

Utilizing 2 ports on NIC, trying to set 1 port to Numa 1, core 0 and Numa 1 core 3.

Using Closest static profile.

Only the first NIC port for Numa 1 core 0 is being "respected.

The second NIC port for Numa 1, core 3 is landing on core 2...

I have tried changing around the second port to be on other cores as well to no avail...

Any idea if there is a bug to numa node base processor core binding?

Each NIC set to max process or 1 and queue 1...

Any suggestions would be appreciated...

idata · ‎09-19-2017

Hi Tilla,

Thank you for posting in Wired Communities. Can you provide the tool use to check on the status of the RSS output or where did you see the NIC port that core 3 is landed on Core 2? If possible can you share the steps on how you configure?

Thanks,

Sharon

RTill · ‎09-21-2017

Thank you so much for responding, I appreciate any help that you can offer...

This is a bit long, but I will give you as much info on the physical setup and app as I can.

Physical Hardware Setup:

So I have a 2 physical processor HP Z840.

I have an X710-DA4 in PCIe slot # 3, which is "attached" to CPU1.

I have a device that streams heavy UDP data over 1-N of the ports on the X710-DA4.

I want each port of the 710-DA4 to "land" on a different core of CPU1.

Application:

I have an application that reads the streams that I have affinitize to CPU0 (this is to observe the receive load placed on CPU1 by the socket guaranteeing that the load is not from the app).

I then initiate a data stream from the device to my application and observe CPU load on task manager.

NIC Setup, Test # 1:

Preferred NUMA Node: 1

Max Num RSS Processors: 1

Max Num RSS Queues: 1

RSS: enabled

RSS Base Processor Number: 0

RSS Load Balancing profile: ClosestProcessorStatic

NIC Setup, Test # 2:

Preferred NUMA Node: 1

Max Num RSS Processors: 1

Max Num RSS Queues: 1

RSS: enabled

RSS Base Processor Number: 1

RSS Load Balancing profile: ClosestProcessorStatic

Expectations:

In the graphs above, CPU0 is the top 4 graphs, CPU1 is the bottom 4 graphs.

By changing the base processor number, I expected the socket receive CPU load to shift from processor 0 to processor 1 on CPU# 1.

Nothing else is running on the machine.

As you can see form the graphs, the recv socket load did not move to CPU1 from CPU0 despite the change in configuration...

Thanks for your time...

RTill · ‎09-21-2017

I have tried many different permutations of configs as well, perhaps there is something about the config that I am not understanding or interpreting properly...

idata · ‎09-21-2017

Hi Tilla,

Thank you for the detail information. I will check on this then update you.

Regards,

Sharon

idata · ‎09-28-2017

Hi Tilla,

Can you provide the output of Get-NetAdapterRss for test# 1 and test# 2 ? Thank you.

Regards,

Sharon

idata · ‎10-08-2017

Hi Tilla,

Please feel free to provide the information requested for me to better investigate.

Thanks,

Sharon

RTill · ‎10-16-2017

Sorry for late response, I was on vaca last week...

The box has gone to the field, so I do not have direct access to it any more

I will see if I can find another box to recreate.

idata · ‎10-16-2017

Hi Tilla,

Thank you for the update and hope you had a great vacation. Feel free to update me when you find another system for the setup.

Thanks,

Sharon

idata · ‎10-24-2017

Hi Tilla,

Please feel free to update me in case you already have the setup.

Thanks,

Sharon

RTill · ‎11-08-2017

Test # 1:

Name : 10GB-1-10.1

InterfaceDescription : Intel(R) Ethernet Converged Network Adapter X710

Enabled : True

NumberOfReceiveQueues : 1

Profile : ClosestStatic

BaseProcessor: [Group:Number] : :0

MaxProcessor: [Group:Number] : :

MaxProcessors : 1

RssProcessorArray: [Group:Number/NUMA Distance] :

IndirectionTable: [Group:Number] :

Test # 2

Name : 10GB-1-10.1

InterfaceDescription : Intel(R) Ethernet Converged Network Adapter X710

Enabled : True

NumberOfReceiveQueues : 1

Profile : ClosestStatic

BaseProcessor: [Group:Number] : :1

MaxProcessor: [Group:Number] : :

MaxProcessors : 1

RssProcessorArray: [Group:Number/NUMA Distance] :

IndirectionTable: [Group:Number] :

idata · ‎11-08-2017

Hi Tilla,

Thank you for the information. I will further check on this.

Regards,

Sharon

idata · ‎11-14-2017

Hi Tilla,

Further checking, please set Max Num RSS Processors = 2 and "RSS Base Processor Number" to 1.

Please feel free to update me.

Thanks,

Sharon

RTill · ‎11-15-2017

I could try that, but in essence isnt that saying "use either core 1 or core 2" (base processor=1 and max processors = 2)?

I desire to be able to pin CPU load for a specific NIC port to a single specific processor in order to efficiently load balance my high network traffic.

idata · ‎11-15-2017

Hi Tilla,

Thank you for the reply. If you set "Max Num RSS Processors" to 1, it only allows RSS to use CPU 0. Setting "Max Num RSS Processors" to 2 and "RSS Base Processor Number" to 1 would force all to CPU 1.

Max processors = 1 can use only CPU 0

Max processors = 2 can use CPU 0 and 1

Max processors = 3 can use CPU 0,1,2

Max processors = 4 can use CPU 0,1,2,3

Please feel free to update me.

Thanks,

Sharon

RTill · ‎11-16-2017

Ahhh, OK, thanks for clearing up my confusion

idata · ‎11-16-2017

Hi Tilla,

You are welcome and please do let me know if further assistance needed.

Thanks,

Sharon