Failover not working on X520 NIC with ixgbevf driver

[GORaw1]

I have the following setup:

• Virtual environment with Openstack with Intel X520 NIC
• Hypervisor using ixgbe driver
• Virtual machine using ixgbevf driver (version 4.6.1) on Red Hat Linux 7.6
• VM interfaces are bonded in active-standby mode on ingress and egress

In normal state everything is fine, the bond interfaces are operational, e.g.:

$ more /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth2

MII Status: up

MII Polling Interval (ms): 0

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: eth2

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: fa:16:3e:07:55:40

Slave queue ID: 0

Slave Interface: eth3

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: fa:16:3e:7c:dd:ab

Slave queue ID: 0

However when the primary interface ens2f0 is taken down on the hypervisor, failover does not occur to the eth3 interface on the VM and traffic fails:

$ more /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: eth2

MII Status: up

MII Polling Interval (ms): 0

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: eth2

MII Status: up

Speed: Unknown

Duplex: Unknown

Link Failure Count: 0

Permanent HW addr: fa:16:3e:07:55:40

Slave queue ID: 0

Slave Interface: eth3

MII Status: up

Speed: 10000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: fa:16:3e:7c:dd:ab

Slave queue ID: 0

eth2 is still marked as UP by the bonding module and as the currently active slave even though the link is down:

$ ethtool eth2 | grep detected

Link detected: no

There aren't any errors logged on either VM or hypervisor, except for:

On the VM:

kernel: ixgbevf 0000:00:06.0 eth2: NIC Link is Down

kernel: ixgbevf 0000:00:08.0 eth4: NIC Link is Down

On the hypervisor

kernel: ixgbe 0000:37:00.0: removed PHC on ens2f0

When the ens2f0 interface on the hypervisor is brought up again, everything is fine, i.e. the bond appears to work only if the original primary interface is running.

Are there any issues with this driver in this setup (though it is the latest driver version), or could there be a configuration which is not applied?

Thanks

Greg

[Crisselle_Intel]

Hello Greg,

Thank you for posting in Intel Ethernet Communities. 

Kindly provide the following details for us to check on your query.

1.) Exact model of the adapter. Please also provide the PBA number for us to double check on it. You may refer to the link below on where to find the PBA.

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-io/ethernet-products.html

2.) Please share the ixgbe driver version used and kernel version information

Looking forward to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[GORaw1]

Hi Criselle,

Adapter model is:

Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

Subsystem: Intel Corporation Ethernet Server Adapter X520-2

PBA: E68793-006

ixgbe driver version:  5.6.3

Hypervisor:

Red Hat 7.6

kernel version: 3.10.0-957.5.1

VM:

Red Hat 7.5

kernel version: 3.10.0-862.11.6

Thanks

Greg

[Crisselle_Intel]

Hello Greg,

Thank you for providing the requested details.

After checking, it is best to try the channel bonding configuration in kernel version 4.x and up with latest driver version as there are many features from kernel 3.10 that has been backported by Red Hat which hardly resemble the actual versions release from kernel.org.

Let us know if you have additional questions and clarifications on this request.

Looking forward to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Crisselle_Intel]

Hello Greg,

Good day!

Please let us know if Failover is now working with your setup. We would highly appreciate if you can send us an update for the status of this request.

We hope to hear from you soon.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[GORaw1]

Hi Crisselle,

Do you mean upgrading the kernel to version 4.x or later? That is not an option with this particular deployment.

Does bonding and failover not work with the ixgbe/ixgbevf driver for Red Hat kernel versions 3.x?

Thanks

Greg

[Crisselle_Intel]

Hello Greg,

Yes. We hope you don't mind sharing more details why upgrading the kernel to 4.x and later is not an option with this particular deployment. This would help us to better understand and further check this request.

Awaiting to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Crisselle_Intel]

Hello Greg,

Good day!

We'd like to follow the requested details above for us to forward it to our engineering team and further check on this request.

Looking forward to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[GORaw1]

Hi - this is for a particular customer deployment which is on Red Hat kernel 3.x and upgrading the kernel version is not possible as it is tied to versions of other software.

Can you confirm if failover is supported for this NIC card on the existing kernel version?

Thanks

Greg

[Crisselle_Intel]

Hello Greg,

We appreciate your prompt reply.

Please allow us to further check on this. Rest assured that we will get back to you within 1-3 business days.

Hoping for your patience.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Crisselle_Intel]

Hello Greg,

Good day!

Please be informed that we are still looking into your request. We hope you don't mind providing additional information below which will be helpful for us to further check.

1.) Do you have any FCOE configuration enabled? Disabling FCOE stuff could fix some bonding issues.

2.) Have you tried posting on Linux/Sourceforge forums regarding this issue?

We look forward to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Crisselle_Intel]

Hello Greg,

I hope you're doing great!

We'd like to check if you have Red Hat account? If yes, kindly visit the link below that might be helpful to your issue resolution.

https://access.redhat.com/solutions/27863

Looking forward to hear an update from you.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[GORaw1]

Hi - thanks for the reply. I will check this link and see if it is helpful.

Thanks

Greg

[Crisselle_Intel]

Hello Greg,

You are welcome, appreciate your response. 

We hope that the provided link will be helpful to this issue. We will wait for your update once you have checked the link above.

Looking forward to hear from you.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Crisselle_Intel]

Hello Greg,

Good day!

We'd like to check if you have visited the link that we provided. We would highly appreciate if you can share an update for the status of this request.

Awaiting to your response.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Crisselle_Intel]

Hello Greg,

How are you today?

Please be informed that we will now close this request since we haven't received any response from our previous follow ups.  Should you have any other concerns or assistance needed in the future, please do not hesitate to post a new question.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel