Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4864 Discussions

SRIOV + ubuntu bonding in KVM

psund2
Beginner
‎09-11-2018 11:56 AM
3,081 Views

00:04.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

I have a KVM with Intel SRIOV card. 5: enp4s0f0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:0c:bd:05:d9:82 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:0c:bd:05:d9:11, spoof checking off, link-state auto vf 1 MAC 00:0c:bd:05:d9:ab, spoof checking off, link-state auto vf 2 MAC 00:0c:bd:05:d9:a2, spoof checking off, link-state auto vf 3 MAC 00:0c:bd:05:d9:ac, spoof checking off, link-state auto vf 4 MAC 00:0c:bd:05:d9:ad, spoof checking off, link-state auto vf 5 MAC 00:0c:bd:05:d9:ae, spoof checking off, link-state auto vf 6 MAC 00:0c:bd:05:d9:af, spoof checking off, link-state auto vf 7 MAC 00:0c:bd:05:d9:a1, spoof checking off, link-state auto 8: enp4s0f1: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:0c:bd:05:d9:83 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:0c:bd:05:d9:12, spoof checking off, link-state auto vf 1 MAC 00:0c:bd:05:d9:bb, spoof checking off, link-state auto vf 2 MAC 00:0c:bd:05:d9:bc, spoof checking off, link-state auto vf 3 MAC 00:0c:bd:05:d9:bd, spoof checking off, link-state auto vf 4 MAC 00:0c:bd:05:d9:be, spoof checking off, link-state auto vf 5 MAC 00:0c:bd:05:d9:bf, spoof checking off, link-state auto vf 6 MAC 00:0c:bd:05:d9:b1, spoof checking off, link-state auto vf 7 MAC 00:0c:bd:05:d9:b2, spoof checking off, link-state auto

I have used vf0 from each card in a GUEST Ubuntu machine

Physical Function enp4s0f0 has the following virtual functions: PCI BDF Interface ======= ========= 0000:04:10.0 0000:04:10.2 0000:04:10.4 0000:04:10.6 0000:04:11.0 0000:04:11.2 0000:04:11.4 0000:04:11.6

Physical Function enp4s0f1 has the following virtual functions: PCI BDF Interface ======= ========= 0000:04:10.1 0000:04:10.3 0000:04:10.5 0000:04:10.7 0000:04:11.1 0000:04:11.3 0000:04:11.5

Snippet of dumpxml

My bonding configuration:

uto eth1 iface eth1 inet manual bond-master bond0

auto eth2 iface eth2 inet manual bond-master bond0

auto bond0 iface bond0 inet static address 192.168.23.101 netmask 255.255.255.0 bond-slaves none bond-mode 2 bond-miimon 100 bond-downdelay 0 bond-updelay 0

cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (xor) Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0

Slave Interface: eth1 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:bd:05:d9:11 Slave queue ID: 0

Slave Interface: eth2 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:bd:05:d9:12 Slave queue ID: 0

bond0 Link encap:Ethernet HWaddr 00:0c:bd:05:d9:11 inet addr:192.168.23.101 Bcast:192.168.23.255 Mask:255.255.255.0 inet6 addr: fe80::20c:bdff:fe05:d911/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:122 errors:0 dropped:0 overruns:0 frame:0 TX packets:295 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:33881 (33.8 KB) TX bytes:34294 (34.2 KB)

eth1 Link encap:Ethernet HWaddr 00:0c:bd:05:d9:11 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:122 errors:0 dropped:0 overruns:0 frame:0 TX packets:183 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:33881 (33.8 KB) TX bytes:28212 (28.2 KB)

eth2 Link encap:Ethernet HWaddr 00:0c:bd:05:d9:11 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:112 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:6082 (6.0 KB)

Here bond0 MAC is same as eth1 MAC as eth1 came up first. Issues is when I receive packets in PF corresponding to eth2 , packets are recieved in GUEST(eth2) , eth2 RX is 0 ...

What could be the issue..sometimes eth2 comes up first after reboot and bond0 has eth2 MAC and at that point eth1 RX is zero..

0 Kudos

Link Copied

9 Replies
idata
Employee
‎09-12-2018 09:35 PM
1,411 Views

Hello Premnathsundarababu,

 

 

Thank you for posting in Intel Wired Ethernet Communities.
  • Please identify the Ubuntu version of the host system.
  • What device are the ports connecting to?
  • If this is a switch does it support 802.3ad, and is trunking or etherchannel enabled on the ports?
  • Please remove one of the cables to test if fault tolerance is working.
  • If spanning tree protocol is enabled this may also cause issues.

If you have any questions please do not hesitate to ask.

Best regards,

 

Daniel D
0 Kudos
Copy link
psund2
Beginner
‎09-13-2018 07:26 AM
1,411 Views
In response to idata

Hi Daniel,

Thanks for your response :

  • Please identify the Ubuntu version of the host system.

I am using ubuntu "14.04"

  • What device are the ports connecting to?

Its a cisco switch

If this is a switch does it support 802.3ad, and is trunking or etherchannel enabled on the ports?

I am using static port-channel

interface GigabitEthernet1/0/1

switchport access vlan 1010

switchport mode access

channel-group 1 mode on

spanning-tree portfast

!

interface GigabitEthernet1/0/2

switchport access vlan 1010

switchport mode access

channel-group 1 mode on

spanning-tree portfast

!

interface Port-channel1

switchport access vlan 1010

switchport mode access

and xor bonding(type 2) in ubuntu

  • Please remove one of the cables to test if fault tolerance is working.

No.Even if i shut the working port in switch..I dont see any packets in ubuntu interface.

  • If spanning tree protocol is enabled this may also cause issues.

I am sure STP is switch side is not an issue here.

Spanning tree enabled protocol ieee

Root ID Priority 33778

Address e804.6271.8180

This bridge is the root

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 33778 (priority 32768 sys-id-ext 1010)

Address e804.6271.8180

Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Aging Time 300 sec

Interface Role Sts Cost Prio.Nbr Type

------------------- ---- --- --------- -------- --------------------------------

Gi1/0/9 Desg FWD 4 128.9 P2p

Gi1/0/10 Desg FWD 4 128.10 P2p

Po1 Desg FWD 3 128.488 P2p

Po46 Desg FWD 3 128.848 P2p

In response to idata
0 Kudos
Copy link
idata
Employee
‎09-13-2018 08:10 PM
1,411 Views
In response to psund2

Hello Premnathsundarababu,

 

 

Thank you for the reply. Please provide an ethtool -i output for the host Ethernet interface. We will investigate the issue and update you as soon as possible. Please let us know if you have any questions.

 

 

Best regards,

 

Daniel D
In response to psund2
0 Kudos
Copy link
idata
Employee
‎09-18-2018 06:19 PM
1,411 Views
In response to psund2

Hello Premnathsundarababu,

 

 

Please indicate the vendor of this adapter or the board if the controller is integrated into a motherboard. ethtool -i output for the interface would also be helpful to see the driver and firmware being used. Please let us know if you have any questions.

 

 

Best regards,

 

Daniel D
In response to psund2
0 Kudos
Copy link
psund2
Beginner
‎09-21-2018 02:27 AM
1,411 Views
In response to psund2

Hi,

ethtool -i enp4s0f0

driver: ixgbe

version: 4.2.1-k

firmware-version: 0x1bab0001

expansion-rom-version:

bus-info: 0000:04:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

ethtool -i enp4s0f1

driver: ixgbe

version: 4.2.1-k

firmware-version: 0x1bab0001

expansion-rom-version:

bus-info: 0000:04:00.1

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: no

00:04.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

00:05.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

In response to psund2
0 Kudos
Copy link
idata
Employee
‎09-23-2018 08:51 PM
1,411 Views
In response to psund2

Hello Premnathsundarababu,

 

 

Thank you for the output. We will continue to investigate this issue, and update you as soon as possible. Please let us know if you have any questions.

 

 

Best regards,

Daniel D

In response to psund2
0 Kudos
Copy link
idata
Employee
‎09-26-2018 05:11 PM
1,411 Views
In response to psund2

Hello Premnathsundarababu,

 

 

Please confirm if you have tested the bond in balance-xor without SR-IOV and KVM. Does the bond work properly prior to implementing SR-IOV and KVM? If you have any questions please let us know.

 

 

Best regards,

 

Daniel D

 

Intel Customer Support
In response to psund2
0 Kudos
Copy link
idata
Employee
‎10-01-2018 05:58 PM
1,411 Views
In response to psund2

Hello Premnathsundarababu,

 

 

Please let us know if you still need assistance with this issue. We just want to know if the bond is working without SR-IOV and KVM. Let us know if you have any questions.

 

 

Best regards,

 

Daniel D

 

Intel Customer Support
In response to psund2
0 Kudos
Copy link
idata
Employee
‎10-07-2018 11:11 PM
1,411 Views
In response to psund2

Hello Premnathsundarababu,

 

 

If you have any questions please let us know.

 

 

Best regards,

 

Daniel D

 

Intel Customer Support
In response to psund2
0 Kudos
Copy link
Reply

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.