round-robin bond configuration issue with SR-IOV on intel XXV710 network card

[DNico6]

Hello,

we are curently facing an issue to implement round-robin bond configuration in our VM with SR-IOV and XXV710 network card.

On the compute which hosting the VM, there are 2 intel XXV710 network cards (2 ports for each card)

So, there are 4PF. And for each PF, there are 4 VF as the bellow output is showing : 

9: enp94s0f0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc mq portid 3cfdfeca9c84 state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9c:84 brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC fa:16:3e:20:db:58, spoof checking on, link-state auto, trust off

10: enp94s0f1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9000 qdisc mq portid 3cfdfeca9c85 state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9c:85 brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC fa:16:3e:5e:26:c7, spoof checking on, link-state auto, trust off

11: enp216s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq portid 3cfdfeca9fbc state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9f:bc brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

12: enp216s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq portid 3cfdfeca9fbd state UP mode DEFAULT qlen 1000

  link/ether 3c:fd:fe:ca:9f:bd brd ff:ff:ff:ff:ff:ff

  vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

  vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto, trust off

For the test, our VM is configured with one VF of enp94s0f0 PF and one VF of enp94s0f1 PF

Bellow is the bond configuration down on the VM : 

# cat ifcfg-eth1 

DEVICE=eth1

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

USERCTL=no

SLAVE=yes

NM_CONTROLLED=no

cat ifcfg-eth2

DEVICE=eth2

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

USERCTL=no

NM_CONTROLLED=no

SLAVE=yes

cat ifcfg-bond0

DEVICE=bond0

ONBOOT=yes

BOOTPROTO=static

USERCTL=no

NM_CONTROLLED=no

BONDING_OPTS="mode=0 miimon=100 fail_over_mac=active"

cat ifcfg-bond0.3304

DEVICE=bond0.3304

NAME=bond0.3304

BOOTPROTO=none

ONPARENT=yes

IPADDR=X.X.X.X

NETMASK=255.255.255.0

GATEWAY=X.X.X.X

VLAN=yes

NM_CONTROLLED=no

And when the command systemctl restart network is launched ... Only one of the two interfaces is enslaved into the bond (not always the same, sometimes eth1, sometimes eth2 ...)

# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: eth2

MII Status: up

Speed: 40000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: fa:16:3e:5e:26:c7

Slave queue ID: 0

For information, we have tried the following commands in the VM (but the result is the same)

ip link set eth2 promisc off

ip link set eth1 promisc off

We have also tried to play with trust / state / promisc / spoofchk / vlan parameter ) as showing the following commands on the compute which hosting the VM (but the result is the same)

for i in 0 1 2 3; do ip link set enp94s0f[0..1] vf $i trust on/off ; done

for i in 0 1 2 3 ; do ip link set enp94s0f[0..1] vf $i state enable/auto; done

for i in 0 1 2 3 ; do ip link set enp94s0f[0..1] vf $i promisc off/on; done

for i in 0 1 2 3 ; do ip link set enp94s0f1 vf $i vlan 0 qos 0 ; done

For information, implementation of an active-backup bond configuration (mode=1) works successully

[Caguicla_Intel]

Hello Geroges,

We'd like to follow up the requested information for us to further check on your request.

Awaiting to your reply.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Geroges,

Good day!

Please be informed that we will now proceed closing this request since we haven't receive any response from our previous follow up. Should you need further assistance on this matter, please provide the details that we requested or you may post a new question.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[gdamien]

Hello Crisselle,

Sorry for the late response.

The new versions of the drivers have resolved partially the issue.

Bond is well started with the 2 VF interfaces but some tests are failing.

We are continuing the tests, I'll get back to you as soon as we finish them to give you a more detailed status.

Regards,

Georges

[Caguicla_Intel]

Hello Geroges,

Not a problem at all.

We will wait for your update on the next following days. Just in case we don't hear from you , we'll follow up on September 12. 

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Geroges,

We would highly appreciate if you can share the results of your testing using the new driver versions.

Hoping to hear from you soon.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Geroges,

Good day!

Please be informed that we will now proceed closing this inquiry since we haven't receive any response from you. Should you have any other concerns or assistance needed in the future, feel free to post a new question.

Best regards,

Crisselle C

Intel Customer Support

A Contingent Worker at Intel