Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

round-robin bond configuration issue with SR-IOV on intel XXV710 network card

DNico6
Beginner
2,798 Views

Hello,

 

we are curently facing an issue to implement round-robin bond configuration in our VM with SR-IOV and XXV710 network card.

 

On the compute which hosting the VM, there are 2 intel XXV710 network cards (2 ports for each card)

So, there are 4PF. And for each PF, there are 4 VF as the bellow output is showing : 

 

9: enp94s0f0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc mq portid 3cfdfeca9c84 state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9c:84 brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC fa:16:3e:20:db:58, spoof checking on, link-state auto, trust off

10: enp94s0f1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc mq portid 3cfdfeca9c85 state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9c:85 brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC fa:16:3e:5e:26:c7, spoof checking on, link-state auto, trust off

11: enp216s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 3cfdfeca9fbc state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9f:bc brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

12: enp216s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq portid 3cfdfeca9fbd state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9f:bd brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

 

For the test, our VM is configured with one VF of enp94s0f0 PF and one VF of enp94s0f1 PF

Bellow is the bond configuration down on the VM : 

# cat ifcfg-eth1 

DEVICE=eth1

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

USERCTL=no

SLAVE=yes

NM_CONTROLLED=no

 

cat ifcfg-eth2

DEVICE=eth2

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

USERCTL=no

NM_CONTROLLED=no

SLAVE=yes

 

cat ifcfg-bond0

DEVICE=bond0

ONBOOT=yes

BOOTPROTO=static

USERCTL=no

NM_CONTROLLED=no

BONDING_OPTS="mode=0 miimon=100 fail_over_mac=active"

 

cat ifcfg-bond0.3304

DEVICE=bond0.3304

NAME=bond0.3304

BOOTPROTO=none

ONPARENT=yes

IPADDR=X.X.X.X

NETMASK=255.255.255.0

GATEWAY=X.X.X.X

VLAN=yes

NM_CONTROLLED=no

 

And when the command systemctl restart network is launched ... Only one of the two interfaces is enslaved into the bond (not always the same, sometimes eth1, sometimes eth2 ...)

 

# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

 

Bonding Mode: load balancing (round-robin)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

 

Slave Interface: eth2

MII Status: up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: fa:16:3e:5e:26:c7

Slave queue ID: 0

 

 

For information, we have tried the following commands in the VM (but the result is the same)

ip link set eth2 promisc off

ip link set eth1 promisc off

 

We have also tried to play with trust / state / promisc / spoofchk / vlan parameter ) as showing the following commands on the compute which hosting the VM (but the result is the same)

for i in 0 1 2 3; do ip link set enp94s0f[0..1] vf $i trust on/off ; done

for i in 0 1 2 3 ; do ip link set enp94s0f[0..1] vf $i state enable/auto; done

for i in 0 1 2 3 ; do ip link set enp94s0f[0..1] vf $i promisc off/on; done

for i in 0 1 2 3 ; do ip link set enp94s0f1 vf $i vlan 0 qos 0 ; done

 

 

 

For information, implementation of an active-backup bond configuration (mode=1) works successully

0 Kudos
26 Replies
Caguicla_Intel
Moderator
2,143 Views

Hello DNico6, 

 

Thank you for posting in Intel Ethernet Communities. 

 

Please give us time to check on your query. We will give you an update within 2-3 business days.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello DNico6, 

 

Thank you for the patience on this matter.

 

Kindly share the driver version used on the PF, VF and Linux OS with kernel information for us to further check on your request.

 

Looking forward to your reply.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
DNico6
Beginner
2,143 Views

Dear Crisselle ,

 

Please find below the requested informations.

 

driver version ==>

rpm -qa | grep i40e

i40e-2.7.12-1.x86_64

 

cat /etc/redhat-release 

Red Hat Enterprise Linux Server release 7.4 (Maipo)

 

uname -a

Linux <hostname>3.10.0-693.35.1.el7.x86_64 #1 SMP Thu Jun 7 07:29:04 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

 

Regards,

Nicolas DEROUIN

0 Kudos
Mike_Intel
Moderator
2,143 Views

Hello DNico6, 

 

Thank you for providing the driver version.

 By the way, can you confirm if you are using on-board NIC or PCIe NIC?

Looking forward to your reply.

 

Best regards,

Michael L.

Intel Customer Support

A Contingent Worker at Intel

 

0 Kudos
DNico6
Beginner
2,143 Views

Dear Michael,

 

Thanks for your reply.

We are using PCIe NIC.

 

Regards,

Nicolas DEROUIN

0 Kudos
Mike_Intel
Moderator
2,143 Views

Hello DNico6, 

 

Thank you for the confirmation. Can you send photos of both sides of the NIC focusing on the markings?

Looking forward to your reply.

 

Best regards,

Michael L.

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
DNico6
Beginner
2,143 Views

Dear Mikel,

 

Please find attached the photos :)

 

Regards,

Nicolas DEROUIN

0 Kudos
DNico6
Beginner
2,143 Views
posted a file.
0 Kudos
Mike_Intel
Moderator
2,143 Views

Hello DNico6, 

 

I tried validating the markings of the network card that you provided and found that this card

is not a retail Intel network card and this card belongs to Cisco. Have you tried checking the issue with Cisco or have you tried to ask them if the have an updated driver for Linux?

If you have questions, please let us know.

 

Best regards,

Michael L.

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello Nicolas, 

 

While we are still looking into your query, we'd like to check if you have tried contacting Cisco to check if they have an updated driver for Linux.

 

Awaiting to your reply.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello Nicolas, 

 

We apologize for the delay on this matter.

 

Based on the details you have provided, it seems that VF driver version is missing. If the virtual function (VF) driver is lower than 3.7.53, please try the latest version on the link below:

https://downloadcenter.intel.com/download/24693/Intel-Network-Adapter-Virtual-Function-Driver-for-Intel-Ethernet-Controller-700-Series

 

Same with the PF (i40e), try out the latest version 2.9.21 and let us know if updating to latest version was of help to your issue

https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=95261

 

Looking forward to your reply.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
gdamien
Beginner
2,143 Views

Hello Crisselle,

Thank you for your support.

We are actually running RHEL 7.4 kernel version 3.10.0-693.35.1.el7.x86_64

I cannot find any information about kernel version requirement in the notice of the VF driver version 3.7.53 : https://downloadmirror.intel.com/24693/eng/readme.txt

Would please confirm that the version 3.7.53 is compatible with this kernel version ?

 

Regards,

Georges DAMIEN

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello Geroges,

 

We appreciate your reply.

 

Please allow us to check on this further and we'll give you an update within 1-3 business days.

 

Hoping for your patience.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello Geroges,

 

We apologize for the delay on this matter.

 

Please be informed that we are still looking into your query. Rest assured that we will provide an update within 1-3 business days.

 

We appreciate your patience.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello Geroges,

 

Thank you for the patience on this matter.

 

After further checking, it is best to try the channel bonding configuration in kernel version 4.x and up with i40e driver (2.9.21) and iavf driver (3.7.53) as there are many features from kernel 3.10 that has been backported by Red Hat which hardly resemble the actual versions release from kernel.org.

 

Let us know if you have additional questions and clarifications.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello Geroges,

 

We'd like to check if you were able to try the latest i40e driver (2.9.21) and iavf driver (3.7.53). Kindly share the results after trying it out.

 

Looking forward to your response.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
gdamien
Beginner
2,143 Views

Hello,

 

I've tested the iavf driver 3.7.53 on rhel7.4 (kernel=3.10.0-693.el7.x86_64).

After the driver installation, the VF interfaces disappeared from the virtual machine.

I guesss it's due to driver/kernel incompatibility.

I'm trying to find out which iavf driver version works with 3.10.0 kernel.

 

Regards,

Georges

0 Kudos
Caguicla_Intel
Moderator
2,143 Views

Hello Geroges,

 

Thank you for sharing your observation.

 

We will wait for your update on this matter. But as mentioned above, it is best to try the channel bonding configuration in kernel version 4.x and up with i40e driver (2.9.21) and iavf driver (3.7.53). 

 

We hope you can find iavf driver version that works with 3.10.0 kernel.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
gdamien
Beginner
2,143 Views

Hello Crisselle,

 

I have few question about driver versions :

Is there any correlation between  the i40e driver version and iavf driver version on both the host and the VM (compatibility matrix)?

Is the iavf driver version has any importance on the host, or what matters is the version inside the hosted virtual machine ?

 

Regards,

Georges

0 Kudos
Caguicla_Intel
Moderator
2,058 Views

Hello Geroges,

 

Thank you for the reply.

 

1.) Kindly provide the link for compatibility matrix link that you mentioned for us to look into your query.

2.) Here is our answer for your question number 2.

Whenever virtual functions are created in the host, Virtual Function drivers are loaded. Therefore the VF driver version is also important to the host as updated VF drivers incorporate fixes and additional support.

 

For sample change log, please refer to the link below:

https://sourceforge.net/projects/e1000/files/i40evf%20stable/3.6.10/

 

Looking forward to hear from you.

 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

0 Kudos
Reply