Issue with setting smp_affinity on ixgbe cards

[KNimb]

Hi,

I am using a Dell PowerEdge R730 with Dual Xeon, each 22 cores, with 6 ixgbe compatible cards, on which I am running Linux with ixgbe driver version 4.4.0-k, using kernel versions both 4.7.10 and 4.9.6.

I am loading the ixgbe modules at boot time, bringing up the interfaces and setting smp_affinity to the cards, using the set_irq_affinity script, so all the possible RxTx IRQs are distributed between all the available cores. The problem is that it happens, random, but quite often that the smp_affinity setting fails, and I need manually to re-run the script one or more times in order desired settings to be applied. There were also several occasions when the settings were not applied at all, and it took me several reboots to script to start working again. The problem appears not only randomly as occurrence, but also at random NIC controllers, so I am excluding the possibility of failed HW, since I also changed NICs.

I added some debug messages to track the affinity setting in Linux kernel, and it turns out that most of the times when the setting fails the error that affinity setting function irq_do_set_affinity returns is EBUSY, but also sometimes it returns ENOSPC.

More investigation on the topic showed whenever EBUSY was returned the problem could be overcome with re-running the script. But if the error returned was ENOSPC, it takes several reboots for the problem to disappear.

In order to provide some more details on the system I am attaching two text files with the output of the modinfo of the ixgbe and lspci on the machine.

[idata]

Hi KM29,

What is the exact model of the ixgbe compatible cards?Thanks,wb

[KNimb]

Hi wb,

Thanks for the swift response .

In one of the occasion I am using Silicom's PE310G6SPI9 Six port Fiber 10Gbit 82559ES Based card. In other occasions I am using Hotlava's Tambora 80G4S-G3 4 port 10Gbit with a 2 port Intel's Ethernet Converged Network Adapter X520-SR2. As I wrote in the initial mail I rotated different configurations in order to isolate the problem to a specifc card , but though it kept appearing with all configurations.

Cheers,

Kula Nimbus

[idata]

HI KM29,

Thank you for the information. I will check if we have any information to share. Thanks,wb

[idata]

Hi KM29,

Please try use driver version 5.0.4 at https://sourceforge.net/projects/e1000/files/ixgbe%20stable/ You can also contact RHEL support to double check.rgds,wb

[KNimb]

Hi wb,

Thanks for checking this once again. I just downloaded the version and built it on my kernel and got pretty much the same result. From the very first attempt my affinity settings were ignored and for almost all the NICs I needed to re-appy it couple of times in order they to take affect.

Here is the modinfo now:

# modinfo ixgbe

filename: /lib/modules/4.9.6-mine/updates/drivers/net/ethernet/intel/ixgbe/ixgbe.ko

version: 5.0.4

license: GPL

description: Intel(R) 10GbE PCI Express Linux Network Driver

author: Intel Corporation, <</span>mailto:linux.nics@intel.com linux.nics@intel.com>

srcversion: 342373434C53E2CDAA7E0DF

alias: pci:v00008086d000015E5sv*sd*bc*sc*i*

alias: pci:v00008086d000015E4sv*sd*bc*sc*i*

alias: pci:v00008086d000015CEsv*sd*bc*sc*i*

alias: pci:v00008086d000015CCsv*sd*bc*sc*i*

alias: pci:v00008086d000015CAsv*sd*bc*sc*i*

alias: pci:v00008086d000015C8sv*sd*bc*sc*i*

alias: pci:v00008086d000015C7sv*sd*bc*sc*i*

alias: pci:v00008086d000015C6sv*sd*bc*sc*i*

alias: pci:v00008086d000015C4sv*sd*bc*sc*i*

alias: pci:v00008086d000015C3sv*sd*bc*sc*i*

alias: pci:v00008086d000015C2sv*sd*bc*sc*i*

alias: pci:v00008086d000015ADsv*sd*bc*sc*i*

alias: pci:v00008086d000015ACsv*sd*bc*sc*i*

alias: pci:v00008086d000015ABsv*sd*bc*sc*i*

alias: pci:v00008086d000015B0sv*sd*bc*sc*i*

alias: pci:v00008086d000015AAsv*sd*bc*sc*i*

alias: pci:v00008086d000015D1sv*sd*bc*sc*i*

alias: pci:v00008086d00001563sv*sd*bc*sc*i*

alias: pci:v00008086d00001560sv*sd*bc*sc*i*

alias: pci:v00008086d00001558sv*sd*bc*sc*i*

alias: pci:v00008086d0000154Asv*sd*bc*sc*i*

alias: pci:v00008086d00001557sv*sd*bc*sc*i*

alias: pci:v00008086d0000154Fsv*sd*bc*sc*i*

alias: pci:v00008086d0000154Dsv*sd*bc*sc*i*

alias: pci:v00008086d00001528sv*sd*bc*sc*i*

alias: pci:v00008086d000010F8sv*sd*bc*sc*i*

alias: pci:v00008086d0000151Csv*sd*bc*sc*i*

alias: pci:v00008086d00001529sv*sd*bc*sc*i*

alias: pci:v00008086d0000152Asv*sd*bc*sc*i*

alias: pci:v00008086d000010F9sv*sd*bc*sc*i*

alias: pci:v00008086d00001514sv*sd*bc*sc*i*

alias: pci:v00008086d00001507sv*sd*bc*sc*i*

alias: pci:v00008086d000010FBsv*sd*bc*sc*i*

alias: pci:v00008086d00001517sv*sd*bc*sc*i*

alias: pci:v00008086d000010FCsv*sd*bc*sc*i*

alias: pci:v00008086d000010F7sv*sd*bc*sc*i*

alias: pci:v00008086d00001508sv*sd*bc*sc*i*

alias: pci:v00008086d000010DBsv*sd*bc*sc*i*

alias: pci:v00008086d000010F4sv*sd*bc*sc*i*

alias: pci:v00008086d000010E1sv*sd*bc*sc*i*

alias: pci:v00008086d000010F1sv*sd*bc*sc*i*

alias: pci:v00008086d000010ECsv*sd*bc*sc*i*

alias: pci:v00008086d000010DDsv*sd*bc*sc*i*

alias: pci:v00008086d0000150Bsv*sd*bc*sc*i*

alias: pci:v00008086d000010C8sv*sd*bc*sc*i*

alias: pci:v00008086d000010C7sv*sd*bc*sc*i*

alias: pci:v00008086d000010C6sv*sd*bc*sc*i*

alias: pci:v00008086d000010B6sv*sd*bc*sc*i*

depends:

vermagic: 4.9.6-mine SMP mod_unload

parm: EEE:Energy Efficient Ethernet (EEE) ,0=disabled, 1=enabled )default EEE disable (array of int)

parm: InterruptType:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default IntMode (deprecated) (array of int)

parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)

parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)

parm: DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)

parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)

parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable (1 queue) 2-16 enable (default=8) (array of int)

parm: max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)

parm: VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)

parm: InterruptThrottleRate:Maximum interrupts per second, per vector, (0,1,956-488281), default 1 (array of int)

parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)

parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)

parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)

parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)

parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)

parm: FdirPballoc:Flow Director packet buffer allocation level:

1 = 8k hash filters or 2k perfect filters

2 = 16k hash filters or 4k perfect filters

3 = 32k hash filters or 8k perfect filters (array of int)

parm: AtrSampleRate:Software ATR Tx packet sample rate (array of int)

parm: MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int)

parm: LRO:Large Receive Offload (0,1), default 0 = off (array of int)

parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599 based adapters, default 0 = Disable (array of int)

parm: dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)

parm: vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)

[idata]

Hi KM29,

Have you contacted RHEL support ?Thanks,wb

[KNimb]

Hi wb,

I have posted same question on the sourceforge https://sourceforge.net/p/e1000/bugs/559/ Intel Ethernet Drivers and Utilities / Bugs / # 559 Issue with setting smp_affinity on ixgbe cards

Since I am not using Red Hat, I have no argument to post at RHEL support site, so I don't see why I should do it, and I did not post anything there.

Best Regards,

Kula Nimbus

[idata]

Hi KM29,

Thank you for the update. Have you tried turning off the irqbalance daemon? Thanks,wb

[KNimb]

Hi wb,

Since my target is manual handling of affinity, I don't even have the irqbalance installed.

So I guess it removes it out of the equation.

Best Regards,

Kula Nimbus

[idata]

Hi KM29,

Please continue to monitor the sourceforge website for updates on upcoming fixes.

Thanks,

wb

[KNimb]

Hi wb,

Thanks for replying. Current stage at sourceforge, is that the issue was closed with the explanation that an internal bug was opened, but I did not get any details how to track the "internal bug".

Best Regards,

Kula Nimbus

[idata]

Hi KM29,

Thanks for the update.Rgds,wb

[KNimb]

Hi wb,

I have small update on this issue. Since the issue on sourceforge got closed, I will update only this thread for now.

I think it may be interesting for you or give a major clue. I was able to reproduce the same issue also with an i40e controller (Intel Corporation Ethernet Controller X710) on the same platform I posted originally.

Best Regards,

Kula Nimbus

[idata]

Hi KM29,

Thank you for the update. Just to double check what is the driver you used here? rgds,wb

[KNimb]

Hi wb,

I the version before the last of the i40e which is 2.0.19. I saw today that there is version 2.0.23 released last week, but I saw no changes that might be related to similar issue.

Here is my modinfo:

filename: /lib/modules/4.9.16-mine/updates/drivers/net/ethernet/intel/i40e/i40e.koversion: 2.0.19license: GPLdescription:Intel(R) 40-10 Gigabit Ethernet Connection Network Driverauthor: Intel Corporation, <</span>mailto:e1000-devel@lists.sourceforge.net e1000-devel@lists.sourceforge.net>srcversion: BE776B7F0BA28FF4DCEE168alias: pci:v00008086d0000158Bsv*sd*bc*sc*i*alias: pci:v00008086d0000158Asv*sd*bc*sc*i*alias: pci:v00008086d000037D3sv*sd*bc*sc*i*alias: pci:v00008086d000037D2sv*sd*bc*sc*i*alias: pci:v00008086d000037D1sv*sd*bc*sc*i*alias: pci:v00008086d000037D0sv*sd*bc*sc*i*alias: pci:v00008086d000037CFsv*sd*bc*sc*i*alias: pci:v00008086d000037CEsv*sd*bc*sc*i*alias: pci:v00008086d0000374Csv*sd*bc*sc*i*alias: pci:v00008086d00001588sv*sd*bc*sc*i*alias: pci:v00008086d00001587sv*sd*bc*sc*i*alias: pci:v00008086d00001589sv*sd*bc*sc*i*alias: pci:v00008086d00001586sv*sd*bc*sc*i*alias: pci:v00008086d00001585sv*sd*bc*sc*i*alias: pci:v00008086d00001584sv*sd*bc*sc*i*alias: pci:v00008086d00001583sv*sd*bc*sc*i*alias: pci:v00008086d00001581sv*sd*bc*sc*i*alias: pci:v00008086d00001580sv*sd*bc*sc*i*alias: pci:v00008086d00001574sv*sd*bc*sc*i*alias: pci:v00008086d00001572sv*sd*bc*sc*i*depends: vermagic: 4.9.16-mine SMP mod_unloadparm: debug:Debug level (0=none,...,16=all) (int)parm: ix_crcstrip:intparm: ix_rx_miss:intparm: ix_rx_miss_bufs:int

Best Regards,

Kula Nimbus

[idata]

HI KM29,

Thank you for the information. rgds,wb

[idata]

HI KM29,

We have informed the engineer and please continue to check with Ubuntu support. Thanks,wb

[KNimb]

Hi wb,

Thank you for the feedback, but maybe there is some misunderstanding here. I am not using Ubuntu, why do you think I should check with Ubuntu support?

Is there a similar issue reported for Ubuntu, that your engineers are handling and updating Ubuntu support site?

If yes, can you please point me to that thread?

If not, maybe it is better to track this issue here where all the details are. Can I rely that once there is some feedback from the engineering department you will update this thread?

Best Regards,

Kula Nimbus

[idata]

Hi KM29,

Thank you for the update and clarification. I will further check then update you.Rgds,wb

[idata]

HI KM29,

Please help provide the exact Linux distribution.Thanks,wb