Multicast join using multiple VM with Intel X520-DA2, 82599 Ethernet Control VF

[DButl3]

I'm trying out mutlicast streaming of uncompressed HD and UHD video uisng VMs and SRIOV with an Intel X520-DA2 NIC (82599 Etherent Control VF).

I can send a mutlicast stream from a VM OK and receive a mutlicast stream on a VM using a SRIOV VF. I'm not using VLANs, just separate MAC/IP address on the same LAN.

If I have 2 VMs on the same host, using separate SRIOV VFs from the same Intel NIC, I can see multicast traffic on both VM interfaces (when only 1 VM has joined the stream).

I have tried turning allmutlicast & promisc mode on/off, but it seems to make no difference.

I can see the traffic using the network stats and using tcpdump in the unjoined VM.

Is this a feature of the NIC, or is there a way to stop traffic going to both VFs/VMs.

For the purpose of testing, I'm using KVM (Virtual Machine Manager) on Ubuntu 16.04, my setup is as follows:

$ lsb_release -a

Ubuntu 16.04.3 LTS, xenial

$ uname -a

Linux ap-z420-0 4.4.0-104-generic # 127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ lspci -nn | grep -i 'Ethernet Controller'

06:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

06:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01)

07:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)

07:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)

07:10.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)

07:10.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)

# modinfo ixgbevf

filename: /lib/modules/4.4.0-104-generic/updates/dkms/ixgbevf.koversion: 4.2.2license: GPLdescription: Intel(R) 10 Gigabit Virtual Function Network Driverauthor: Intel Corporation, <</span>mailto:linux.nics@intel.com linux.nics@intel.com>srcversion: 0F89E29947DD944E62FA118

$ ip link show ens3f0

5: ens3f0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 90:e2:ba:72:70:e0 brd ff:ff:ff:ff:ff:ff vf 0 MAC 52:54:00:1d:e1:aa, spoof checking on, link-state auto vf 1 MAC 52:54:00:1d:e1:ab, spoof checking on, link-state auto

Any suggestions gratefully received.

[idata]

Hi Davidpb_RD,

Thank you for posting in Wired Communities. I will further check on this.Regards,Sharon T

[idata]

Hi Davidpb_RD,

Please try use below driver to see if this will work for your setup.https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/ https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/ Feel free to update me.Thanks,Sharon T

[DButl3]

Hi Sharon,

Thanks for the reply.

How related are the ixgbe and ixgbevf drivers?

My colleagues are uisng netmap (kernel stack bypass) to achieve UHD bit rates on a Linux server. The ixgbe driver is patched with netmap specifc code:

https://github.com/luigirizzo/netmap/tree/master/LINUX/final-patches https://github.com/luigirizzo/netmap/tree/master/LINUX/final-patches

Currently we are using ixgbe-5.2.4 and ixgebvf-4.2.2. It is my understanding that ixgbevf-4.2.2 is unmodified.

Previously I had updated to ixgbevf-4.3.2, but had compatability issues.

Does ixgbevf-4.3.3 require ixgbe-5.3.5 or can it be used with an older ixgbe driver?

Thanks,

Dave B.

[idata]

Hi Davidpb_RD,

Thank you for the reply. The ixgbevf driver is for the virtual function (VF) for the virtual NIC. The virtual function driver rely on the services of the physical driver to handle the operation with global impact. It can only be activated on kernels that support SR-IOV.It is recommended to upgrade to latest driver version.Please feel free to update me the result if you proceed to upgrade the driver.Thanks,Sharon T

[DButl3]

Sharon,

i've update to ixgbevf 4.3.3:

$ lsmod | grep ixg

ixgbevf 65536 0

ixgbe_bbcrd 323584 0

vxlan 49152 1 ixgbe_bbcrd

dca 16384 2 ixgbe_bbcrd,ioatdma

netmap 151552 1 ixgbe_bbcrd

ptp 20480 5 ixgbe_bbcrd,e1000e,userclock

$ lsmod | grep ixg

ipstudio-admin@ap-z420-0:~$ modinfo ixgbevf

filename: /lib/modules/4.4.0-104-generic/updates/dkms/ixgbevf.ko

version: 4.3.3

license: GPL

description: Intel(R) 10 Gigabit Virtual Function Network Driver

author: Intel Corporation, <</span>mailto:linux.nics@intel.com linux.nics@intel.com>

srcversion: 2054440C35DA63B66C62480

Again I am running 2 Ubuntu 16.04 VMs, using VMM (KVM) on an Ubuntu 16.04 host with sriov, where both VF are on same physical interface (ens3f0).

$ ip link show ens3f0

5: ens3f0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 90:e2:ba:72:70:e0 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 52:54:00:1d:e1:aa, spoof checking on, link-state auto

vf 1 MAC 52:54:00:1d:e1:ab, spoof checking on, link-state auto

When 1 VM joins the uncompressed HD multicast stream, again I see 1.09Gb/s on the sriov VF network interfaces in both VMs.

Dave B.

[idata]

Hi Davidpb_RD,

Thank you for the update. Let me further check.Regards,Sharon T

[idata]

Hi Davidpb_RD,

Further checking, the multicast in SR-IOV is working as designed. X520 has a limited number of Multicast filters in SR-IOV. When the virtual function multicast is enabled in SR-IOV mode, all the VF created will see the multicast traffic. This can be avoided if you can enable the VLAN tags for the virtual functions.Thanks,Sharon T

[DButl3]

Sharon,

I tried adding VLAN tags to each VF, e.g.

$ ip link show ens3f0

5: ens3f0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 90:e2:ba:72:70:e0 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 52:54:00:1d:e1:aa, vlan 100, spoof checking on, link-state auto

vf 1 MAC 52:54:00:1d:e1:ab, vlan 200, spoof checking on, link-state auto

However, this broke the network connectivity. The VM interface coudl not get an IP address via DHCP

My VMs need to be on the same LAN / subnet and the connection to the host server is not a trunk connection.

Is this what you meant by enabling VLAN tags on the VFs?

I only need 2 to 4 VMs, Is there a way to use whatever mutlicast filtering is available in SRIOV mode?

I have not found any info on doing this.

Thanks,

Dave.

[idata]

Hi Davidpb_RD,

Thank you for the update. Let me further check.Regards,Sharon T

[idata]

Hi Davidpb_RD,

The VLAN configuration should be done before the VF driver is loaded or the VM is booted. Please try to check the sample command below: $ ip link set dev vf vlan For example, the following instructions will configure PF eth0 and the first VF on VLAN 10.$ ip link set dev eth0 vf 0 vlan 10Thanks,Sharon T

[DButl3]

Sharon,

Those are the commands I used to set a vlan for each VF.

My understanding is that using separate VLANs for each VF on the same PF also requires the VLANs to be configured in the rest of the network.

This will make the network routing of the mutlicast streams significantly more complex and could cause duplication of streams.

This approach is not really compatible with how our network is set up.

Is there another way of stopping the mutlicast stream going to un-joined VMs?

There are non-Intel NICs that handle mutlicast streams correctly, but I also want rate control per VF.

If the X520 cannot do this , are there other 10G/25G Intel NICs that support netmap, multicast, PF rate control and VF rate control?

Thanks,

Dave.

[idata]

Hi Davidpb_RD,

Thank you for the reply. Let me further check.Regards,Sharon T

[idata]

Hi Davidpb_RD,

As mentioned earlier this is working as designed. In case you are interested, you may look at the Fortville adapters: https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ethernet-x710-brief.pdf As stated in the product brief " VMDq for Emulated Path: VMDq, enables a hypervisor to represent a single network port as multiple network ports that can be assigned to the individual VMs. Traffic handling is offloaded to the network controller, delivering the benefits of port partitioning with little to no administrative overhead by the IT staff." "SR-IOV for Direct Assignment: Adapter-based isolation and switching for various virtual station instances enables optimal CPU usage in virtualized environments.• With up to 128 Virtual Functions (VF)s, each VF can support a unique and separate data path for I/Orelated functions within the PCI Express* hierarchy.• Use of SR-IOV with a networking device, for example, allows the bandwidth of a single port (function) to be partitioned into smaller slices that can be allocated to specific VMs or guests, via a standard interface. Hope the above information help.Regards,Sharon T

[idata]

Hi David_RD,

Please feel free to update me if there are other inquiries. Thank you.Regards,Sharon T