Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4811 Discussions

Issue with setting smp_affinity on ixgbe cards

KNimb
Novice
6,393 Views

Hi,

I am using a Dell PowerEdge R730 with Dual Xeon, each 22 cores, with 6 ixgbe compatible cards, on which I am running Linux with ixgbe driver version 4.4.0-k, using kernel versions both 4.7.10 and 4.9.6.

 

I am loading the ixgbe modules at boot time, bringing up the interfaces and setting smp_affinity to the cards, using the set_irq_affinity script, so all the possible RxTx IRQs are distributed between all the available cores.

 

The problem is that it happens, random, but quite often that the smp_affinity setting fails, and I need manually to re-run the script one or more times in order desired settings to be applied. There were also several occasions when the settings were not applied at all, and it took me several reboots to script to start working again.

 

The problem appears not only randomly as occurrence, but also at random NIC controllers, so I am excluding the possibility of failed HW, since I also changed NICs.

I added some debug messages to track the affinity setting in Linux kernel, and it turns out that most of the times when the setting fails the error that affinity setting function irq_do_set_affinity returns is EBUSY, but also sometimes it returns ENOSPC.

More investigation on the topic showed whenever EBUSY was returned the problem could be overcome with re-running the script. But if the error returned was ENOSPC, it takes several reboots for the problem to disappear.

In order to provide some more details on the system I am attaching two text files with the output of the modinfo of the ixgbe and lspci on the machine.

0 Kudos
37 Replies
idata
Employee
711 Views

HI KM29,

 

 

Just want to follow up the exact Linux distribution.

 

 

Thanks,

 

wb
0 Kudos
KNimb
Novice
711 Views

Hi wb,

Sorry for not answering that earlier, I was offline for several days.

On the question, I am using Gentoo, but my kernels are directly from https://www.kernel.org/ The Linux Kernel Archives and the drivers for the cards are from https://sourceforge.net/projects/e1000/ Intel Ethernet Drivers and Utilities download | SourceForge.net , that's why I thought the distribution should not matter.

Best Regards,

Kula Nimbus

0 Kudos
idata
Employee
711 Views

HI KM29,

 

 

Thank you for the information.

 

 

rgds,

 

wb

 

0 Kudos
idata
Employee
711 Views

Hi KM29,

 

 

If you are using a new enough kernel then you should not have to use the driver

 

from sourceforge or downloadcenter.

 

 

I understand you are using kernel version 4.7.10 and 4.9.6. Are both kernel version obtained from Gentoo website? Can you provide the link?

 

 

 

It would be better you get better information directly from the kernel mailing list if the issue is kernel related.

 

 

 

Thanks,

 

wb

 

0 Kudos
KNimb
Novice
711 Views

Hi wb,

Thanks for the reply.

If I understand correctly, you recommend not to use ixgbe and i40e drivers from sourceforge, but to use those from kernel.org, although those are a bit older, and with less capabilities?

Is this a general recommendation or you would like me to check if this might be caused by the differences between the drivers from sourceforge and in-tree ones?

Here is a link to the https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tag/?h=v4.9-rc6 4.9 kernel I reproduced it with.

Also if I understood correctly if I reproduce it with the in-kernel drivers I should raise the issue in the kernel mailing list ?

And in case it does not happen with those drivers to return to you?

Best Regards,

Kula Nimbus

0 Kudos
idata
Employee
711 Views

Hi KM29,

 

 

Based from the kernel link you've shared, it's a release candidate version 4.9.rc6. Is it possible to try out the latest release kernel - version 4.11.4?

 

 

 

Best Regards,

 

Vince

 

 

 

0 Kudos
idata
Employee
711 Views

Hi KM29,

 

 

Please let us know if you were able to try out the latest release kernel - version 4.11.4. Thanks.

 

 

Best regards,

 

Vince
0 Kudos
idata
Employee
711 Views

Hi KM29,

 

 

We'd like to check if you still need assistance regarding X520.

 

 

regards,

 

Vince
0 Kudos
idata
Employee
711 Views

HI KM29,

Can you share what is the serial# of the X520-SR2? In case it is convenient for you, please send the screenshot of the network adapter bearing the serial number, here is the sample format of the serial #

https://www.intel.com/content/www/us/en/support/boards-and-kits/000005625.html https://www.intel.com/content/www/us/en/support/boards-and-kits/000005625.html

Looking forward to your reply.

Thanks,

sharon

0 Kudos
KNimb
Novice
711 Views

Hi,

I can provide with a sample serial # of a X520 this happened with: 001B21А48588101АDЕ66560-004

Still I wish to point your attention to that, as I wrote previously, this problem reproduced with a wide variety of NICs, which rules out the options this is specific to a certain NICs - amongst the NICs this happend with were:

- Silicom's PE310G6SPI9 Six port Fiber 10Gbit (82599 Based)

- Hotlava's Tambora 80G4S-G3 4 port 10Gbit (82599 Based)

- 2 port Intel's Ethernet Converged Network Adapter X520-SR2 - tried several different ones

It also reproduced with a wide variety of servers, so it's not specific to a certain machine.

Thanks,

KM

0 Kudos
idata
Employee
711 Views

Hi KM,

 

 

Thank you for the additional information provided. We will check on this.

 

 

regards,

 

sharon

 

0 Kudos
KNimb
Novice
711 Views

Hi Sharon,

Do you know by any chance of any progress on this issue?

I see it is marked as "answered", though it is not, so just checking up if there is some update on it.

Best Regards,

Kula Nimbus

0 Kudos
KNimb
Novice
711 Views

Hello, intel_corp? Is there anybody there?

The issue still exists, over a month passed with no response - the thread somehow got marked as answered.

Can you say if this was looked at internally by the R&D? is this an open issue/case? will this be handled?

0 Kudos
idata
Employee
711 Views

Hi KM29,

 

 

Apologize for the delay for any inconvenience this might caused. Further checking as this occur on multiple adapters, have you tried checking with Linux support as this likely something to do with the configuration.

 

 

Thanks,

 

Sharon
0 Kudos
idata
Employee
711 Views

Hi KM29,

 

 

Have you tried checking with Linux support for the configuration?

 

 

Thanks,

 

Sharon

 

 

0 Kudos
KNimb
Novice
711 Views

Hi Sharon,

Can you recommend a configuration, in which the problem was tested and did not appear, so I can try to recreate it on my setup?

Thanks in advance.

Best Regards,

Kula Nimbus

0 Kudos
idata
Employee
711 Views

Hi KM29,

 

 

We recommend you to check with Linux as this is likely something specific with the configuration at your end. Thus you need to work with Linux. Hope this clarifies.

 

 

Thanks,

 

Sharon

 

0 Kudos
Reply