Problems with X710 VFs behind PCIe switch when using Multiqueue

[StefanKrupop]

Hi!

We have built a special microserver platform where one has multiple COM Express modules on carriers. The carriers also include an X710 NIC chip to allow the CXP modules 10 GBit Network access. As there is only one NIC for three CXP modules, we use the SR-IOV functionality of the X710 in conjunction with a special PCIe switch (PLX9749) to instantiate two VFs for each CXP module and forward them towards the modules, so that they just see the VFs.

To make matters even more complex, the CPU managing the carrier and PLX switch, and thus loading the PF driver, is an ARM processor (while the CXP modules are standard x86).

After many struggles we got the setup basically working, however only when forcing the VF driver to just use one Rx/Tx queue.

When I try to enable two or more queues (I added an module parameter to the iavf driver to be able to quickly do so), I very soon get a "Transmit queue timeout".

Now I am wondering where this could come from. Where is the difference when using multiple queues above just one? All the basic principles (descriptor ring list, DMA, MSI-X interrupts etc.) seem to be the same independent of the number of queues, just more...

I also once tried to have multiple queues, but just 2 MSI-X interrupts. That didn't help either.

Does anyone have insight into what else changes when using multiqueue? Are the PCIe read/write requests larger? Does the HMC work differently with multiple queues?

I will post the dmesg output of one vs. two enabled queues in the next post, it makes this one to long to post ;-)

Thank you!

Kind regards,

Stefan

[StefanKrupop]

dmesg of CXP module with one queue:

[ 210.580485] iavf: loading out-of-tree module taints kernel. [ 210.580533] iavf: module verification failed: signature and/or required key missing - tainting kernel [ 210.581360] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver - version 3.7.61.20 [ 210.581360] Copyright(c) 2013 - 2019 Intel Corporation. [ 210.581506] iavf 0000:03:02.2: enabling device (0000 -> 0002) [ 210.583257] iavf 0000:03:0a.2: enabling device (0000 -> 0002) [ 210.671200] iavf 0000:03:0a.2: Invalid MAC address 00:00:00:00:00:00, using random [ 210.671209] iavf 0000:03:0a.2: Forcing number of active queues to 1 [ 210.678267] iavf 0000:03:0a.2: Requested 2-2 MSI-X interrupts, got 2 [ 210.678279] iavf 0000:03:0a.2: Multiqueue Disabled: Queue pair count = 1 [ 210.678721] iavf 0000:03:0a.2: MAC address: 22:e8:f7:06:65:de [ 210.678725] iavf 0000:03:0a.2: GRO is enabled [ 210.678754] iavf 0000:03:02.2: Invalid MAC address 00:00:00:00:00:00, using random [ 210.678762] iavf 0000:03:02.2: Forcing number of active queues to 1 [ 210.679378] iavf 0000:03:02.2: Requested 2-2 MSI-X interrupts, got 2 [ 210.679383] iavf 0000:03:02.2: Multiqueue Disabled: Queue pair count = 1 [ 210.679778] iavf 0000:03:02.2: MAC address: e2:f0:b7:f6:62:11 [ 210.679782] iavf 0000:03:02.2: GRO is enabled [ 210.683853] iavf 0000:03:0a.2 enp3s10f82: renamed from eth0 [ 210.703547] iavf 0000:03:02.2 enp3s2f18: renamed from eth1 [ 210.813577] iavf 0000:03:0a.2 enp3s10f82: NIC Link is Up Speed is 10 Gbps Full Duplex [ 210.813600] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s10f82: link becomes ready [ 210.837543] iavf 0000:03:02.2 enp3s2f18: NIC Link is Up Speed is 10 Gbps Full Duplex [ 210.837553] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s2f18: link becomes ready

[StefanKrupop]

With two queues enabled:

[ 322.770998] iavf: Intel(R) Ethernet Adaptive Virtual Function Network Driver - version 3.7.61.20 [ 322.770999] Copyright(c) 2013 - 2019 Intel Corporation. [ 322.863553] iavf 0000:03:02.2: Invalid MAC address 00:00:00:00:00:00, using random [ 322.863561] iavf 0000:03:02.2: Forcing number of active queues to 2 [ 322.864162] iavf 0000:03:02.2: Requested 2-3 MSI-X interrupts, got 3 [ 322.864168] iavf 0000:03:02.2: Multiqueue Enabled: Queue pair count = 2 [ 322.864565] iavf 0000:03:02.2: MAC address: 7e:52:61:a9:9c:d7 [ 322.864568] iavf 0000:03:02.2: GRO is enabled [ 322.869987] iavf 0000:03:02.2 enp3s2f18: renamed from eth0 [ 322.870461] iavf 0000:03:0a.2: Invalid MAC address 00:00:00:00:00:00, using random [ 322.870463] iavf 0000:03:0a.2: Forcing number of active queues to 2 [ 322.896347] iavf 0000:03:0a.2: Requested 2-3 MSI-X interrupts, got 3 [ 322.896355] iavf 0000:03:0a.2: Multiqueue Enabled: Queue pair count = 2 [ 322.896781] iavf 0000:03:0a.2: MAC address: f6:fb:9c:e2:fe:75 [ 322.896783] iavf 0000:03:0a.2: GRO is enabled [ 322.901496] iavf 0000:03:0a.2 enp3s10f82: renamed from eth0 [ 322.992257] iavf 0000:03:02.2 enp3s2f18: NIC Link is Up Speed is 10 Gbps Full Duplex [ 322.992266] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s2f18: link becomes ready [ 322.993772] iavf 0000:03:0a.2 enp3s10f82: NIC Link is Up Speed is 10 Gbps Full Duplex [ 322.993780] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s10f82: link becomes ready [ 328.679597] ------------[ cut here ]------------ [ 328.679630] NETDEV WATCHDOG: enp3s10f82 (iavf): transmit queue 1 timed out [ 328.679674] WARNING: CPU: 4 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x258/0x260 [ 328.679676] Modules linked in: iavf(OE) bnep bluetooth ecdh_generic ecc nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua sof_pci_dev snd_sof_intel_hda_common snd_sof_intel_byt snd_sof_intel_ipc snd_sof snd_sof_nocodec snd_sof_xtensa_dsp snd_soc_skl snd_soc_hdac_hda snd_hda_ext_core snd_soc_skl_ipc snd_soc_sst_ipc snd_soc_sst_dsp snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core snd_compress ac97_bus snd_hda_codec_hdmi snd_pcm_dmaengine snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi intel_rapl_msr intel_rapl_common mei_hdcp snd_seq x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel snd_seq_device kvm joydev input_leds snd_timer irqbypass intel_cstate intel_rapl_perf snd wmi_bmof soundcore mei_me intel_pch_thermal mei acpi_pad mac_hid sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq [ 328.679757] libcrc32c raid1 raid0 multipath linear hid_logitech_hidpp hid_logitech_dj crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i915 i2c_algo_bit aesni_intel drm_kms_helper syscopyarea sysfillrect sysimgblt aes_x86_64 crypto_simd fb_sys_fops cryptd nvme glue_helper sdhci_pci e1000e cqhci drm nvme_core ahci i2c_i801 sdhci libahci wmi video pinctrl_cannonlake pinctrl_intel hid_generic usbhid hid [last unloaded: iavf] [ 328.679794] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G OE 5.3.0-18-generic #19-Ubuntu [ 328.679796] Hardware name: Default string Default string/Default string, BIOS 1.05.10 08/24/2018 [ 328.679802] RIP: 0010:dev_watchdog+0x258/0x260 [ 328.679806] Code: 85 c0 75 e5 eb 9f 4c 89 ff c6 05 ad 56 eb 00 01 e8 0d f9 fa ff 44 89 e9 4c 89 fe 48 c7 c7 d0 ca e0 ae 48 89 c2 e8 03 3f 74 ff <0f> 0b eb 80 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 d7 [ 328.679809] RSP: 0018:ffffb6eec0210e30 EFLAGS: 00010286 [ 328.679812] RAX: 0000000000000000 RBX: ffff9aa5713a8bc0 RCX: 000000000000083f [ 328.679815] RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f [ 328.679817] RBP: ffffb6eec0210e60 R08: ffff9aa57b917448 R09: 0000000000000004 [ 328.679818] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000004 [ 328.679820] R13: 0000000000000001 R14: ffff9aa56903a480 R15: ffff9aa56903a000 [ 328.679823] FS: 0000000000000000(0000) GS:ffff9aa57b900000(0000) knlGS:0000000000000000 [ 328.679826] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 328.679828] CR2: 00007fcfc7189000 CR3: 000000088b40a005 CR4: 00000000003606e0 [ 328.679831] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 328.679833] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 328.679834] Call Trace: [ 328.679837] <IRQ> [ 328.679846] ? pfifo_fast_enqueue+0x150/0x150 [ 328.679854] call_timer_fn+0x32/0x130 [ 328.679860] __run_timers.part.0+0x174/0x280 [ 328.679866] ? tick_sched_handle+0x33/0x60 [ 328.679870] ? tick_sched_timer+0x3d/0x80 [ 328.679877] ? recalibrate_cpu_khz+0x10/0x10 [ 328.679880] ? ktime_get+0x42/0xa0 [ 328.679886] run_timer_softirq+0x2a/0x50 [ 328.679891] __do_softirq+0xe1/0x2d6 [ 328.679897] ? hrtimer_interrupt+0x13b/0x220 [ 328.679902] irq_exit+0xae/0xb0 [ 328.679906] smp_apic_timer_interrupt+0x7b/0x140 [ 328.679910] apic_timer_interrupt+0xf/0x20 [ 328.679912] </IRQ> [ 328.679919] RIP: 0010:cpuidle_enter_state+0xc5/0x420 [ 328.679923] Code: ff e8 ef 8a 83 ff 80 7d c7 00 74 17 9c 58 0f 1f 44 00 00 f6 c4 02 0f 85 3d 03 00 00 31 ff e8 22 e1 89 ff fb 66 0f 1f 44 00 00 <45> 85 ed 0f 89 d1 01 00 00 41 c7 44 24 10 00 00 00 00 48 83 c4 18 [ 328.679925] RSP: 0018:ffffb6eec00ffe38 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13 [ 328.679929] RAX: ffff9aa57b92b340 RBX: ffffffffaf15a700 RCX: 000000000000001f [ 328.679931] RDX: 0000000000000000 RSI: 000000002f32908f RDI: 0000000000000000 [ 328.679933] RBP: ffffb6eec00ffe78 R08: 0000004c86d43c6f R09: 000000000000afc7 [ 328.679935] R10: ffff9aa57b92a0e4 R11: ffff9aa57b92a0c4 R12: ffff9aa57b936900 [ 328.679937] R13: 0000000000000008 R14: 0000000000000008 R15: ffff9aa57b936900 [ 328.679944] ? cpuidle_enter_state+0xa1/0x420 [ 328.679949] cpuidle_enter+0x2e/0x40 [ 328.679953] call_cpuidle+0x23/0x40 [ 328.679957] do_idle+0x1eb/0x280 [ 328.679961] cpu_startup_entry+0x20/0x30 [ 328.679966] start_secondary+0x168/0x1c0 [ 328.679972] secondary_startup_64+0xa4/0xb0 [ 328.679977] ---[ end trace fc1beb19c4c64b69 ]--- [ 328.906539] iavf 0000:03:02.2 enp3s2f18: NIC Link is Up Speed is 10 Gbps Full Duplex [ 329.064414] iavf 0000:03:0a.2 enp3s10f82: NIC Link is Up Speed is 10 Gbps Full Duplex [ 335.048265] iavf 0000:03:02.2 enp3s2f18: NIC Link is Up Speed is 10 Gbps Full Duplex [ 335.056272] iavf 0000:03:0a.2 enp3s10f82: NIC Link is Up Speed is 10 Gbps Full Duplex [ 341.207854] iavf 0000:03:0a.2 enp3s10f82: NIC Link is Up Speed is 10 Gbps Full Duplex [ 345.798861] iavf 0000:03:02.2 enp3s2f18: NIC Link is Up Speed is 10 Gbps Full Duplex

[AlfredoS_Intel]

Hi StefanKrupop,

Thank you for posting in our Intel® Ethernet Communities Forum.

May we know the operating system that you are using, including the distro and version?

Looking forward to your reply. Should we not get your reply, we will follow up with you after 3 business days.

Best Regards,

Alfred S

Intel® Customer Support

[StefanKrupop]

Thank you for your reply.

The x86 host is running Ubuntu 19.10. I downloaded and compiled the IAVF driver version 3.7.61.20 separately.

The ARM controller where the PF driver (i40e) is loaded is running Kernel 4.4.32 and a custom Buildroot distro.

Kind regards,

Stefan

[AlfredoS_Intel]

Hi StefanKrupop,

Thank you for your response.

Kindly also provide us the results of this command: ethtool -i ethx where ethx is the Ethernet port.

We appreciate you cooperation with this matter.

Looking forward to your reply. If we do not hear from you, we will check back after 3 business days.

Best Regards,

Alfred S

Intel® Customer Support

[StefanKrupop]

On the PF (ARM) side:

ethtool -i eth1

driver: i40e

version: 1.3.46-k

firmware-version: 6.80 0x80003cea 0.0.0

expansion-rom-version:

bus-info: 0001:03:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

On the VF (x86) side:

ethtool -i enp3s2f18

driver: iavf

version: 3.7.61.20

firmware-version: N/A

expansion-rom-version:

bus-info: 0000:03:02.2

supports-statistics: yes

supports-test: no

supports-eeprom-access: no

supports-register-dump: no

supports-priv-flags: yes

[AlfredoS_Intel]

Hi StefanKrupop,

Thank you for sharing those information.

Please allow us some time to check on your concern.

We will try to get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi StefanKrupop,

Thank you for waiting for our update.

After our initial investigation, here is our recommendation:

1. Please try to update the firmware of the card. You can find the firmware version from this page, https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-700-Series?product=189530.

After doing that, please test the connection again.

To give you ample time, we will wait for your update for 3 business days. Should we not hear from you, we will check back with you.

Best Regards,

Alfred S

Intel® Customer Support

[StefanKrupop]

Dear Alfred,

I updated the firmware, but sadly my problem did not change. I still get "NETDEV WATCHDOG: enp3s10f82 (iavf): transmit queue 1 timed out" on the VF side.

Ethtool now shows:

ethtool -i eth1

driver: i40e

version: 1.3.46-k

firmware-version: 7.30 0x80008379 0.0.0

expansion-rom-version:

bus-info: 0001:03:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

Can you maybe shine some light on my original questions?

"Does anyone have insight into what else changes when using multiqueue? Are the PCIe read/write requests larger? Does the HMC work differently with multiple queues?"

Thank you!

Kind regards,

Stefan Krupop

[AlfredoS_Intel]

Hi StefanKrupop,

Thank you for trying our recommendation.

We will continue to investigate your concern.

The question that you laid out was already in our agenda when started our investigations.

You are most welcome.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi StefanKrupop,

We are still investigating on your concern.

While we were checking the logs, we have noticed that the versions of the driver listed are not the most updated ones.

These are the things that we noticed:

On the PF (ARM) side:

ethtool -i eth1

driver: i40e

version: 1.3.46-k

On the VF (x86) side:

ethtool -i enp3s2f18

driver: iavf

version: 3.7.61.20

The latest drivers for the items above are:

i40e driver

latest version is 2.11.29

https://sourceforge.net/projects/e1000/files/i40e%20stable/

iavf driver

latest version is 3.9.5

https://sourceforge.net/projects/e1000/files/iavf%20stable/

You mentioned that you have already tried a lot of things, so we just want to check if you have already tried with newer versions of the driver.

[StefanKrupop]

Finally!
After month of debugging this problem, I finally found a solution/workaround.

The thing that finally brought me on the right track was when I noticed, due to added debug output, that queue_mapping[] was 0 for queues 1-3 for each VF, while queue_mapping[0] was set to the correct value. This lead to the PF driver configuring QTX_CTL(0) instead of the correct one when the VF asked for it, probably causing the errors I saw on the PF driver side.

Some of my debug output, showing the "wrong" QTX_CTL offset:
[ 1327.840585] i40e 0001:03:00.0: i40e_vc_config_queues_msg: Setting up 2 queue pairs for VSI 12
[ 1327.849530] i40e 0001:03:00.0: i40e_vc_config_queues_msg: Configuring VSI queue 0
[ 1327.864140] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: Context base=0x184eed00, qlen=512, rdylist=0x6, head_wb_ena=0, head_wb_addr=0x0
[ 1327.893294] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: I40E_QTX_CTL(73) = 0x100
[ 1327.900826] i40e 0001:03:00.0: i40e_vc_config_queues_msg: Configuring VSI queue 1
[ 1327.933264] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: Context base=0x184eed40, qlen=512, rdylist=0x6, head_wb_ena=0, head_wb_addr=0x0
[ 1327.968234] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: I40E_QTX_CTL(0) = 0x100

I then checked why the other queues where not set up and finally found that it was limited by the amount of MSIx-Interrupts assigned, which in turn was limited by there only being one CPU core on the PF side.

My fix was to replace the num_online_cpus() call in i40e_init_msix() with a constant "4" to force pf->num_lan_msix being 4 instead of 1. With that in place in the PF driver, the default iavf driver now finally works on the x86 and the queue does no longer time out.

It seems to me this could be a bug in the PF driver? I checked and found similar code in the current version (as I am using the version in an older Kernel, but also tried before with the current driver). The problem only seems to occur if the PF side has less CPU cores than the VF driver wants to use queues...

Kind regards,
Stefan Krupop

[StefanKrupop]

Finally!
After month of debugging this problem, I finally found a solution/workaround.

The thing that finally brought me on the right track was when I noticed, due to added debug output, that queue_mapping[] was 0 for queues 1-3 for each VF, while queue_mapping[0] was set to the correct value. This lead to the PF driver configuring QTX_CTL(0) instead of the correct one when the VF asked for it, probably causing the errors I saw on the PF driver side.

Here is some of my debug output, showing the wrong I40E_QTX_CTL for queue 1:
[ 1327.840585] i40e 0001:03:00.0: i40e_vc_config_queues_msg: Setting up 2 queue pairs for VSI 12
[ 1327.849530] i40e 0001:03:00.0: i40e_vc_config_queues_msg: Configuring VSI queue 0
[ 1327.864140] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: Context base=0x184eed00, qlen=512, rdylist=0x6, head_wb_ena=0, head_wb_addr=0x0
[ 1327.893294] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: I40E_QTX_CTL(73) = 0x100
[ 1327.900826] i40e 0001:03:00.0: i40e_vc_config_queues_msg: Configuring VSI queue 1
[ 1327.933264] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: Context base=0x184eed40, qlen=512, rdylist=0x6, head_wb_ena=0, head_wb_addr=0x0
[ 1327.968234] i40e 0001:03:00.0: i40e_config_vsi_tx_queue: I40E_QTX_CTL(0) = 0x100

I then checked why the other queues where not set up and finally found that it was limited by the amount of MSIx-Interrupts assigned, which in turn was limited by there only being one CPU core on the PF side.

My fix was to replace the num_online_cpus() call in i40e_init_msix() with a constant "4" to force pf->num_lan_msix being 4 instead of 1. With that in place in the PF driver, the default iavf driver now finally works on the x86 and the queue does no longer time out.

It seems to me this could be a bug in the PF driver? I checked and found similar code in the current version (as I am using the version in an older Kernel, but also tried before with the current driver). The problem only seems to occur if the PF side has less CPU cores than the VF driver wants to use queues...

Kind regards,
Stefan Krupop

[AlfredoS_Intel]

Hi StefanKrupop,

Thank you for sharing the brilliant workaround that you have found for the issue.

It will be helpful to another community members who may stumble on the same issue that you have.

Please let us know if you have any further questions.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi StefanKrupop,

We are just following up to check if you have any further questions.

We will follow up again after 3 business days. Should we not hear from you, our system may automatically close the thread.

Best Regards,

Alfred S

Intel® Customer Support 

[AlfredoS_Intel]

Hi Stefankrupop, 

We need to close this thread since we have not gotten a response from you: maybe because you are busy or preoccupied at the moment. We know that this is important for you to get it resolved and it is also equally important for us to give you the right solution; as much as we would like to assist you, we need to close it to attend to other customers. We hope for your consideration and understanding on this one.

If you still need assistance with your concern, you can reply back to the thread or create another topic, and we will be more than glad to assist you 

Thank you for contacting Intel® and have a great week!

Best Regards,

Alfred S

Intel® Customer Support

[StrongXiong]

Hi Stefan,

 I meat an issue about set iavf driver support more queue. Currently, it always only 1 queue:

6.856543] iavf 4ec9:00:02.0: Multiqueue Disabled: Queue pair count = 1

And I tried some parameters, but failed:

696.356199] iavf: unknown parameter 'QueuePairs' ignored

Could you tell me how to enable two or more queues? 

Thanks

Strong