ixgbe vf clear to send when PF down

[Ghalem]

Hi everyone,

We encounter an issue with ixgbe driver on 82599ES 10GB SFI/SFP+ Network Connection series.

Here the host configuration:

root@platane:~# lspci | grep Ether
02:00.0 Ethernet controller: Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller
02:00.1 Ethernet controller: Broadcom Limited BCM57414 NetXtreme-E 10Gb/25Gb RDMA Ethernet Controller (rev 01)
03:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
03:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
83:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
83:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
83:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
83:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
root@platane:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3f0np0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:0a:f7:b6:e3:d0 brd ff:ff:ff:ff:ff:ff
3: ens3f1np1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:0a:f7:b6:e3:d1 brd ff:ff:ff:ff:ff:ff
5: mgmt0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 00:1e:67:51:19:77 brd ff:ff:ff:ff:ff:ff
7: ens1f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:e7:0c:fa brd ff:ff:ff:ff:ff:ff
8: ens1f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether e4:1d:2d:e7:0c:fb brd ff:ff:ff:ff:ff:ff
9: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:2b:0f:4c brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:09:c0:2b:0f:4c, spoof checking off, link-state auto, trust off, query_rss off
10: ens7f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 90:e2:ba:2b:0f:4d brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:09:c0:2b:0f:4d, spoof checking off, link-state auto, trust off, query_rss off

modinfo ixgbe | more
filename: /lib/modules/4.15.0-128-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko
version: 5.1.0-k
license: GPL
description: Intel(R) 10 Gigabit PCI Express Network Driver
author: Intel Corporation, <linux.nics@intel.com>
srcversion: 8DC39057A28FAAE6DBE6C36

When PF is turned down then up the guest loses connection on vf attached devices. There is no way to recover connection except by resetting the VFs on the guest.

I have seen that on ixgbe_down() we set vfinfo->clear_to_send to 0, but I dont see the reverse action on ixgbe_up.

Is there any reason to force vf->clear_to_send to false ? If so, is something missing to set clear_to_send back to true when the PF is UP ?

Is there any solution to make VFs come up again or some kind of "ip link pf vf <vfid> state enable" like applicable with i40e driver ?

Thanks for your help.

Here the code for ixgbe_down (checked on most versions of ixgbe from 5.0.4 to 5.6.3).

void ixgbe_down(struct ixgbe_adapter *adapter)
{
struct net_device *netdev = adapter->netdev;
struct ixgbe_hw *hw = &adapter->hw;
int i;

/* signal that we are down to the interrupt handler */
if (test_and_set_bit(__IXGBE_DOWN, &adapter->state))
return; /* do nothing if already down */

/* Shut off incoming Tx traffic */
netif_tx_stop_all_queues(netdev);

/* call carrier off first to avoid false dev_watchdog timeouts */
netif_carrier_off(netdev);
netif_tx_disable(netdev);

/* Disable Rx */
ixgbe_disable_rx_queue(adapter);

/* synchronize_rcu() needed for pending XDP buffers to drain */
if (adapter->xdp_ring[0])
synchronize_rcu();

ixgbe_irq_disable(adapter);

ixgbe_napi_disable_all(adapter);

adapter->flags2 &= ~(IXGBE_FLAG2_FDIR_REQUIRES_REINIT);
clear_bit(__IXGBE_RESET_REQUESTED, &adapter->state);
adapter->flags &= ~IXGBE_FLAG_NEED_LINK_UPDATE;

del_timer_sync(&adapter->service_timer);

if (adapter->num_vfs) {
/* Clear EITR Select mapping */
IXGBE_WRITE_REG(&adapter->hw, IXGBE_EITRSEL, 0);

/* Mark all the VFs as inactive */
for (i = 0 ; i < adapter->num_vfs; i++)
adapter->vfinfo[i].clear_to_send = 0;

/* ping all the active vfs to let them know we are going down */
ixgbe_ping_all_vfs(adapter);

/* Disable all VFTE/VFRE TX/RX */
ixgbe_disable_tx_rx(adapter);
}

/* disable transmits in the hardware now that interrupts are off */
ixgbe_disable_tx_queue(adapter);

#ifdef HAVE_PCI_ERS
if (!pci_channel_offline(adapter->pdev))
#endif
ixgbe_reset(adapter);

/* power down the optics for 82599 SFP+ fiber */
if (hw->mac.ops.disable_tx_laser)
hw->mac.ops.disable_tx_laser(hw);

ixgbe_clean_all_tx_rings(adapter);
ixgbe_clean_all_rx_rings(adapter);
}

[Caguicla_Intel]

Hello Ghalem,

Thank you for posting in Intel Ethernet Communities. 

Please provide the following information that would help in checking your request.

1.) Where was the 5.1.0 driver downloaded from?

2.) Have you tried using the latest (5.10.2) PF driver from Intel?

3.) What is the current version of the VF driver and where it was downloaded from?

4.) Kindly confirm that the Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection is embedded on your motherboard. If yes, share the motherboard brand and model.

5.) Exact Operating System used

6.) Is the issue only specific to Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection?

We hope to hear from you soon. 

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Ghalem,

Good day!

We'd like to follow up the information we requested for us to proceed checking your request. If in case you need more time on this, please let us know so we can adjust our follow up date.

Awaiting to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Ghalem,

I hope you are doing great!

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

Best regards,

Crisselle C

Intel® Customer Support