Hi Sharon and Pratik,
I have the exact same need - I have an 82599ES-based SR-IOV setup using a Mirantis Fuel 9.2 OpenStack deployment where I am trying to pass VLAN-tagged packets out of a KVM VM instance with the VLAN kept intact. Just like Pratik, I can work with a situation where either the tag applied by the virtual machine is retained with no tag added to it on egress or (preferably) a situation where the inner tag applied by the VM is preserved and an outer tag is added by the network adapter.
I have a full lab environment set up to test this and I will be compiling both drivers. However, I'm curious if there is a specific fix that addresses this issue and if you have any additional information on what behavior I should expect. Will the ixgbe 5.1 driver and the ixgbevf 4.1 driver allow both of the situations I described, i.e. either one tag or double tags leaving the VM? Thank you for any clarification you can provide,
I have completed compiling both drivers but loading the new modules does not change the behavior; attempting to set a VLAN inside of a VM results in the VLAN being stripped on egress, after which the VLAN associated with the VF is applied as the only tag present. Here is the system configuration I am working with and a synopsis of the basic steps I took:
# uname -a Linux sriov4-cpu1.bpi.ciena.com 4.4.0-62-generic # 83~14.04.1-Ubuntu SMP Wed Jan 18 18:10:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
~# rmmod ixgbe ixgbevf
~# cd /ixgbe/ixgbe-5.2.1/src
~/ixgbe/ixgbe-5.2.1/src# make install
~/ixgbe/ixgbe-5.2.1/src# cd ~/ixgbevf/ixgbevf-4.2.1/src
~/ixgbevf/ixgbevf-4.2.1/src# make install
~/ixgbevf/ixgbevf-4.2.1/src# modprobe ixgbe ixgbevf
~/ixgbevf/ixgbevf-4.2.1/src# update-initramfs -u
~/ixgbevf/ixgbevf-4.2.1/src# ethtool -i enp4s0f0
firmware-version: 0x800007f5, 17.5.10
All steps and commands succeeded as expected and I was able to start a VM (launch a previously created OpenStack instance, i.e. launch a KVM visible with virsh). I saw the following output from dmesg showing the tail of the modprobe initialization:
[12589154.358105] ixgbevf: eth370: ixgbevf_probe: Intel(R) 82599 Virtual Function
[12589154.358110] ixgbevf: eth370: ixgbevf_probe: GRO is enabled
[12589154.358112] ixgbevf: eth370: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
[12589154.358143] pci 0000:83:1f.3: [8086:10ed] type 00 class 0x020000
[12589154.358189] pci 0000:83:1f.3: can't set Max Payload Size to 256; if necessary, use "pci=pcie_bus_safe" and report a bug
[12589154.358550] iommu: Adding device 0000:83:1f.3 to group 426
[12589154.358617] ixgbevf 0000:83:1f.3: enabling device (0000 -> 0002)
[12589154.420133] ixgbe 0000:83:00.1 enp131s0f1: VF Reset msg received from vf 61
[12589154.420472] ixgbe 0000:83:00.1: VF 61 has no MAC address assigned, you may have to assign one manually
[12589154.436496] ixgbevf 0000:83:1f.3: MAC address not assigned by administrator.
[12589154.436501] ixgbevf 0000:83:1f.3: Assigning random MAC address
[12589154.437558] ixgbevf 0000:83:1f.3: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2
[12589154.438125] ixgbevf: eth371: ixgbevf_probe: Intel(R) 82599 Virtual Function
[12589154.438131] ixgbevf: eth371: ixgbevf_probe: GRO is enabled
[12589154.438132] ixgbevf: eth371: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
[12589160.674214] audit: type=1400 audit(1501615158.835:173): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/sbin/" pid=124042 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[12589160.674224] audit: type=1400 audit(1501615158.835:174): apparmor="DENIED" operation="open" profile="/usr/sbin/ntpd" name="/usr/local/bin/" pid=124042 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[12589436.263828] ixgbevf: eth47: ixgbevf_remove: Remove complete
[12589436.283385] ixgbevf: eth25: ixgbevf_remove: Remove complete
[12589436.327460] ixgbe 0000:04:00.0: setting MAC fa:16:39:4c:63:d6 on VF 16
[12589436.327465] ixgbe 0000:04:00.0: Reload the VF driver to make this change effective.
[12589436.327506] ixgbe 0000:04:00.0: Setting VLAN 1807, QOS 0x0 on VF 16
[12589436.353284] ixgbe 0000:04:00.1: setting MAC fa:16:39:26:5e:1c on VF 8
[12589436.353288] ixgbe 0000:04:00.1: Reload the VF driver to make this change effective.
[12589436.353328] ixgbe 0000:04:00.1: Setting VLAN 1807, QOS 0x0 on VF 8
[12589436.607825] audit: type=1400 audit(1501615434.749:175): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1" pid=161335 comm="apparmor_parser"
[12589436.608013] audit: type=1400 audit(1501615434.749:176): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1//qemu_bridge_helper" pid=161335 comm="apparmor_parser"
[12589436.699374] device tap8643a0ca-f4 entered promiscuous mode
[12589436.727574] qbr8643a0ca-f4: port 2(tap8643a0ca-f4) entered forwarding state
[12589436.727583] qbr8643a0ca-f4: port 2(tap8643a0ca-f4) entered forwarding state
[12589436.929559] audit: type=1400 audit(1501615435.069:177): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1" pid=161558 comm="apparmor_parser"
[12589436.963500] audit: type=1400 audit(1501615435.105:178): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-e3675296-f81e-4d61-9280-ce0e7eaee8a1//qemu_bridge_helper" pid=161558 comm="apparmor_parser"
[12589437.857401] vfio-pci 0000:04:14.0: enabling device (0000 -> 0002)
[12589437.961279] vfio-pci 0000:04:12.1: enabling device (0000 -> 0002)<p...
After extensive testing we have determined that the combination of the current driver and firmware for 82599ES chipset adapters do not have the functionality to allow QinQ / double-tagged / stacked VLAN's on egress. This appears to be a hard limitation with no workaround available. If a VLAN is associated with a VF to allow ingress traffic to reach the correct VM it is not possible to egress on that same VF with a different VLAN. The only solution, and it's an insane one, is to associate two VF's for each port (one with a VLAN tag and one without) and only egress on the VF without the VLAN, using bizarre methods such as a fake bonded port or placing the virtual interfaces in a virtual switch with rules dictating where traffic can flow. This is obviously not a viable workaround in a real VNF situation.
Are there *any* Intel chipsets that work properly in this regard? Mellanox has full QinQ / double tag support as noted at https://community.mellanox.com/docs/DOC-2522 HowTo Configure QinQ Encapsulation per VF in Li... | Mellanox Interconnect Community but I'd like to know if there are any Intel chipsets that will work. Thanks in advance,
Thank you for the further update. You are right double VLAN is not supported on 82599ES products and respective drivers. Our Fortville supports double VLAN (aka QnQ) feature in the hardware, but the software does not support it at this time. Double VLAN (aka QnQ) support is expected in a future software release but we don't have timeline yet. Hope this clarified.