Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

82599ES SR-IOV on Ubuntu 14.04 - VF behavior

JHall7
Beginner
3,281 Views

Hi,

This is my first visit/post here to this community so I hope I'm asking the right questions in the right place.

I am using the 82599ES adapter with SR-IOV enabled under Ubuntu 14.04 under kernel 3.13.0-30 but this same behavior was witness under 3.13.0-24. The corresponding VFs are created and visible in 'lspci' and 'ip link show'. I am intending to pass VFs to guest OSs in KVM running libvirt 1.2.2 as 'hostdev' devices in the guest's XML configuration file.

Once the VFs are created, I proceed to set VF parameters such as a MAC and VLAN ID. I believed MACs were to be automatically generated by the driver but perhaps I'm wrong as it doesn't seem to be the case. So in this particular case, I set the VF MAC to the auto generated MAC that libvirt assigns to the interface by editing the guest through virsh edit.

root@ubuntu:~# ip link set eth4 up

root@ubuntu:~# ip link set eth4 vf 0 mac 52:54:00:4d:8b:bc

root@ubuntu:~# ip link set eth4 vf 0 vlan 100

root@ubuntu:~# ip link set eth4 vf 1 mac 52:54:00:5e:3b:e2

root@ubuntu:~# ip link set eth4 vf 1 vlan 200

root@ubuntu:~# ip link set eth4 vf 2 mac 52:54:00:19:e3:f8

root@ubuntu:~# ip link set eth4 vf 2 vlan 300

root@ubuntu:~# ip link set eth4 vf 3 mac 52:54:00:1c:7e:79

root@ubuntu:~# ip link set eth4 vf 3 vlan 100

root@ubuntu:~# ip link set eth4 vf 4 mac 52:54:00:45:c9:7d

root@ubuntu:~# ip link set eth4 vf 4 vlan 200

root@ubuntu:~# ip link set eth4 vf 5 mac 52:54:00:32:13:0e

root@ubuntu:~# ip link set eth4 vf 5 vlan 300

This results in something like the following (dmesg output below also).

root@ubuntu:~# ip link show dev eth4

11: eth4: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 52:54:00:4d:8b:bc, vlan 100, spoof checking on, link-state auto

vf 1 MAC 52:54:00:5e:3b:e2, vlan 200, spoof checking on, link-state auto

vf 2 MAC 52:54:00:19:e3:f8, vlan 300, spoof checking on, link-state auto

vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

Jul 16 10:29:21 ubuntu kernel: [ 8518.503869] ixgbe 0000:81:00.0: registered PHC device on eth4

Jul 16 10:29:21 ubuntu kernel: [ 8518.608585] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready

Jul 16 10:29:21 ubuntu kernel: [ 8518.672821] ixgbe 0000:81:00.0 eth4: detected SFP+: 3

Jul 16 10:29:21 ubuntu kernel: [ 8518.912379] ixgbe 0000:81:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Jul 16 10:29:21 ubuntu kernel: [ 8518.912536] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready

Jul 16 10:29:28 ubuntu kernel: [ 8526.350238] ixgbe 0000:81:00.0: setting MAC 52:54:00:4d:8b:bc on VF 0

Jul 16 10:29:28 ubuntu kernel: [ 8526.350244] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:28 ubuntu kernel: [ 8526.363123] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 0

Jul 16 10:29:28 ubuntu kernel: [ 8526.387115] ixgbe 0000:81:00.0: setting MAC 52:54:00:5e:3b:...

0 Kudos
6 Replies
Patrick_K_Intel1
Employee
1,476 Views

Thanx for visiting. Sorry for the delayed response - was on vacation :-)

I believe the issues is that you are administratively setting the MAC address for the VF's. When you do this, we make the assumption that the administrator does not trust the VMM or the VM to configure the MAC address, all such requests will be rejected.

In general what usually happens is you create a bunch of VF's, your VMM (such as Virt-Manager) will itself assign a MAC address to the VF when it get assigned to a VM and everything is fine - you should even be able to change the MAC from the VM. Though you may need to turn of anti-spoofing.

My expert came back witht the following that I am pasting for your reading pleasure.

Recommended setup steps are:

  1. Blacklist VF driver if VFs are intended to be assigned toVM(s).
  2. Load ixgbe driver with desired number of VFs.
  3. DO NOT assign MAC address at this time. Assigning MAC address to VF from the host means that administrator doesn't trust the VM and VM will not be able to change VF's MAC address.
  4. Assign Port VLAN for each VF using IPROUTE2 utility.
  5. Assign the VF to VM using Linux Virt-Manager or creating/modifying VM configuration XML file.
  6. Bring up VM to confirm connectivity.

We have tested this setup and it works using Ubuntu 14.04 host and guests.

We believe Linux Virt-Manager and KVM are getting confused because user is assigning the MAC addresses from within the host. For Intel 10Gb network drivers this is called an administrative assignment and is a security feature.

Hope this helps!

- Patrick

0 Kudos
JHall7
Beginner
1,476 Views

Hi Patrick,

Thanks so much for your reply. I hope you enjoyed your vacation time and I'm really sorry you have to deal with me upon your return!

So, I wouldn't normally assign MAC addresses to the VFs, I only went that route because of the random behavior of the VFs after starting the VM that the hostdev interfaces were assigned to.

For this reply, I changed my guest from FreeBSD to CentOS to perhaps simplify things but what I was originally seeing and the reason I even set them was because I was arbitrarily getting an all zeros MAC randomly on the VMs interfaces.

First, in response to your recommended steps.

1) ixgbevf is being blacklisted from within /etc/modprobe.d/blacklist.conf

blacklist ixgbevf

2) I am loading ixgbe and VFs from within /etc/modprobe.d/ixgbe.conf

options ixgbe max_vfs=16

options loop max_loop=128

3) Now begging with my new CentOS VM, I have not assigned MAC addresses to the VFs.

root@ubuntu:~# ip link show dev eth4

5: eth4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto

vf 1 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto

vf 2 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

vf 3 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto

vf 4 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto

vf 5 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

4) VLANs are assigned (see output from 3)

5) Here is the applicable snippet from my guest's XML configuration

6) And the result from starting the guest which still reflects bizarre behavior. Notice from the log output below that a MAC was set to VF0 two times, a MAC was set for VF1 but VF2 was never set.

root@ubuntu:~# ip link show dev eth4

5: eth4: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 52:54:00:58:5f:0c, spoof checking on, link-state auto

vf 1 MAC 52:54:00:51:5c:0a, spoof checking on, link-state auto

vf 2 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

vf 3 MAC 00:00:00:00:00:00, vlan 100, spoof checking on, link-state auto

vf 4 MAC 00:00:00:00:00:00, vlan 200, spoof checking on, link-state auto

vf 5 MAC 00:00:00:00:00:00, vlan 300, spoof checking on, link-state auto

vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

Jul 22 12:53:40 ubuntu kernel: [ 3138.012846] pci-stub 0000:81:10.0: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.012936] pci-stub 0000:81:10.2: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.012951] pci-stub 0000:81:10.4: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.012972] pci-stub 0000:81:10.6: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.012988] pci-stub 0000:81:11.0: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013002] pci-stub 0000:81:11.2: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013017] pci-stub 0000:81:11.4: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013031] pci-stub 0000:81:11.6: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013045] pci-stub 0000:81:12.0: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013057] pci-stub 0000:81:12.2: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013068] pci-stub 0000:81:12.4: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013080] pci-stub 0000:81:12.6: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013091] pci-stub 0000:81:13.0: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013102] pci-stub 0000:81:13.2: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013113] pci-stub 0000:81:13.4: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013125] pci-stub 0000:81:13.6: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013136] pci-stub 0000:81:10.1: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013149] pci-stub 0000:81:10.3: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013166] pci-stub 0000:81:10.5: claimed by stub

Jul 22 12:53:40 ubuntu kernel: [ 3138.013178] pci-stub 0000:81:10....

0 Kudos
Patrick_K_Intel1
Employee
1,476 Views

If I'm reading this correctly, I think the problem is due to the fact that you need to bring the interface up after the driver is loaded.

So make sure you have a cable plugged in, then issue:

ip link set eth4 up

this should bring the link up on the PF, then you can assign the VLAN IDs tothe VF, and then assign the VM to the VM and then bring up the VM.

Give that a try.

- Patrick

0 Kudos
JHall7
Beginner
1,476 Views

Thanks once again for replying.

I actually had corrected the link issue prior to your response and now I know it's not the PF state.

I think this actually a number/order issue. Is there a simple way to establish exactly what VF belongs to which PF? Lspci or dmesg output doesn't seem to establish the connection.

In my previous examples, I used pci_0000_81_10_0, 0000_81_10_1 & 0000_81_10_2 but I am finding out that these aren't consecutive VFs of the same PF or something else has gone awry.

What I got from looking at virsh nodedev-dumpxml is that these three addresses match to the following PFs/VFs...

Am I on the right track?

pci_0000_81_10_0Eth4 VF0pci_0000_81_10_1Eth5 VF1pci_0000_81_10_2Eth4 VF2

root@ubuntu:~# virsh nodedev-dumpxml pci_0000_81_10_0

pci_0000_81_10_0

/sys/devices/pci0000:80/0000:80:01.0/0000:81:10.0

pci_0000_80_01_0

pci-stub

0

129

16

0

82599 Ethernet Controller Virtual Function

Intel Corporation

root@ubuntu:~# virsh nodedev-dumpxml pci_0000_81_10_1

pci_0000_81_10_1

/sys/devices/pci0000:80/0000:80:01.0/0000:81:10.1

pci_0000_80_01_0

pci-stub

0

129

16

1

82599 Ethernet Controller Virtual Function

Intel Corporation

root@ubuntu:~# virsh nodedev-dumpxml pci_0000_81_10_2

pci_0000_81_10_2

/sys/devices/pci0000:80/0000:80:01.0/0000:81:10.2

pci_0000_80_01_0

pci-stub

0

129

16

2

82599 Ethernet Controller Virtual Function

Intel Corporation

root@ubuntu:~# lspci | grep 82599

81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.3 Ethernet controlle...

0 Kudos
Patrick_K_Intel1
Employee
1,477 Views

The good ole, which PF does the VF belong to challenge :-)

For a dual port device, even numbered VF's belongs to PF0, and odd numbered VF's belong to PF1.

So...

81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ß BELONG TO PF# 0 i.e. PF 81:00.0

81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ß BELONG TO PF# 1 i.e. PF 81:00.1

81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ß BELONG TO PF# 0 i.e. PF 81:00.0

81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ß BELONG TO PF# 1 i.e. PF 81:00.1

81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ß BELONG TO PF# 0 i.e. PF 81:00.0

81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) ß BELONG TO PF# 1 i.e. PF 81:00.1

Hope that clears it up.

- Patrick

0 Kudos
JHall7
Beginner
1,477 Views

Thanks for the feedback Patrick and clearing that up. It makes sense now. Also, sorry for the late response. I was away for a few days

It's evident that I will need to set per VF vlans in the guest XML files and I'm doing that like this as an example:

This may be outside the scope of this thread, but do you know if nativeMode='untagged' is necessary or will it treat this VF like an access-port connecting to the guest natively using vlan 300 to pass untagged frames?

JC

0 Kudos
Reply