SR-IOV with ixgbe - spoof packets detected

[sfrid]

Hey All,

I have a VM runs on Cisco server that runs KVM with SR-IOV enabled.

4 VFs are attached to this VM and spoof check is off on all of them.

I still get 'spoofed packets detected' warning all the time

"ixgbe 0000::0c:00.0 eth17: 2 Spoofed packets detected"

I read online that spoof detection is enabled by default (on compilation) on ixgbe driver when SR-IOV is active.

Any idea how to overcome this issue?

If any more information needed please let me know.

Thanks,

Shaham

[VincentT_T_Intel]

Hi Shaham, please share the ixgbe driver version and network adapter model involved in your setup. In case you're using bonding on the VMs, there's a workaround posted in this thread - kindly check if it will be helpful.

regards,

Vince

[sfrid]

Hey Vince,

First of all, thanks for the quick response!

My ixgbe driver's version is: 3.15.1-k.

My network adapter is: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection.

In my setup I don't use Bonding, only two VFs of the same PF which are attached to my VM to be used as client & server ports.

Thanks,

Shaham

[SYeo3]

Hi Shaham,

Thank you for providing the details. I will check on this further.

Sincerely,

Sandy

[SYeo3]

Hi Shaham,

Please refer to the guide below:

http://www.intel.eu/content/dam/doc/design-guide/82599-sr-iov-driver-companion-guide.pdf Intel® 82599 SR-IOV Driver Rev 1.00 Driver Companion Guide

See sections

7.2 MAC Anti Spoofing

7.3 VLAN Tag Anti Spoofing

Feel free to contact us again if you have further questions.

Sincerely,

Sandy

[sfrid]

Hey Sandy,

Thanks for the reference. Correct me if I'm wrong but in order to change these MACAS and VLANAS fields (which are mentioned in sections 7.2 & 7.3),

one should recompile the driver, right?

BTW, when I'm not using VLAN tagging it all works just fine and I suspect that the PF's driver doesn't recognize the VLANs that I defined on the VFs and

therefore warns about spoofing.

This is my VFs configuration:

62: rename62: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 00:e0:ed:2c:6c:af brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:00:00:11:11:11, vlan 100, spoof checking off, link-state auto

vf 1 MAC 00:00:00:22:22:22, vlan 200, spoof checking off, link-state auto

vf 2 MAC 00:00:00:00:00:00, spoof checking off, link-state auto

vf 3 MAC 00:00:00:00:00:00, spoof checking off, link-state auto

Am I configuring something wrong? Is there a way to make the PF aware of the VLANs defined on the VFs?

Thanks,

Shaham

[SYeo3]

Hi Shaham,

Thank you for your updates. I'll check this and will back with updates.

Sincerely,

Sandra

[SYeo3]

Hi Shaham,

Yes, you are correct. To change the MACAS and VLANAS, it is necessary to recompile the driver. To further check on your configuration, we would like to request for your system details. Please provide information below:

•Host OS – distro and version number

•Host OS dmesg and Linux kernel log

•Guest OS- distro and version number

•ixgbevf driver version number.

•Guest OS dmesg and Linux kernel log

Sincerely,

Sandy

[sfrid]

Hey Sandy,

Your quick response is very appreciated!

As for the info you requested:

1. Host OS: Ubuntu 14.04.1 LTS

2. Guest OS: Ubuntu 12.04.5 LTS

3. ixgbevf: 2.11.3-k

4. Host & Guest's dmesg & kernel log: http://www.filedropper.com/dmesgkernlog http://www.filedropper.com/dmesgkernlog

If any more info is needed, please let me know.

Thanks,

Shaham

[SYeo3]

Hi Shaham,

Thanks for the details. We'll check on this.

Sincerely,

Sandy

[SYeo3]

Hi Shaham,

Since you are creating VF in the host. Once the VF's are created the Host OS loads the ixgbevf driver automatically.

Once the VF driver is loaded in the Host OS it will claim all the VF that it finds on the PCI bus.

These VFs are not allowed to be assigned to the VM at this point.

So, we believe this is the reason you are experiencing "Spoof Packet Detected" messages.

Please follow the procedure below:

1. Add "blacklist ixgbevf" to /etc/modprobe.d/blacklist.conf file.

2. Load ixgbe driver

3. Create VF using pci sysfs interface.

4. Assign VF to the VM

5. Boot VM

This should address Spoof Packet Detection issue. Please let us know if you need further assistance.

Sincerely,

Sandy

[sfrid]

Hey Sandy,

Thanks again for the quick response!

I followed the steps mentioned above but yet no luck - I still get spoofed packets.

I'll explain exactly what I did step-by-step:

1. edited /etc/modprobe.d/blacklist.conf and added at the end-of-file "blacklist ixgbevf".

2. rebooted the machine.

3. verified ixgbe driver is loaded.

4. I ran "echo 2 > /sys/bus/pci/devices/0000\:88\:10.0/sriov_numvfs" in order to create 2 VFs for the eth device I want to use.

5. verified the two VFs were actually created, 'lspci | grep Eth' gave the following:

root@laphroaig:~# lspci | grep Eth

02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

02:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

02:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

04:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)

04:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 01)

09:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

09:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

0c:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

0c:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

85:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

85:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

88:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

88:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

88:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

88:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

6. verified ixgbevf driver is not loaded.

7. configured VLAN and SPOOF-CHK on the two generated VFs, so 'ip link show' gives the following:

12: eth14: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 00:e0:ed:2c:6c:ae brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:00:00:00:00:00, vlan 100, spoof checking off, link-state auto

vf 1 MAC 00:00:00:00:00:00, vlan 200, spoof checking off, link-state auto

8. defined a new VM and attached these two VFs above to it.

9. ran this VM and tried to run traffic via it, but still same issue. 'ip link show' now gives the following (MAC addresses were assigned automatically):

12: eth14: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 00:e0:ed:2c:6c:ae brd ff:ff:ff:ff:ff:ff

vf 0 MAC 02:09:c0:93:c6:20, vlan 100, spoof checking off, link-state auto

vf 1 MAC 02:09:c0:1c:58:2e, vlan 200, spoof checking off, link-state auto

Am I doing something wrong?

Thanks a lot for your patience!

Shaham

[st4]

HI Shahamf,

Thank you for the update. Let me check on this.

rgds,

wb

[sfrid]

Hey wb,

Is there any update?

Thanks,

Shaham

[SYeo3]

Hi Shaham,

Thanks for writing back. We are still checking on your configuration. Rest assured, we'll update you once we find anything.

Thank you for your patience and understanding.

Sincerely,

Sandy

[st4]

Hi Shahamf,

Good day. As you are using ixgbe and ixgbevf drivers that are distributed inbox with Ubuntu. In box drivers are intended for basic connectivity and are maintained by the distribution publisher which is ubuntu in this case. Please try upgrading to latest ixgbe and ixgbevf drivers. Below are the driver version and download URL for the drivers.

• Ixgbe driver version 4.1.1 -

http://sourceforge.net/projects/e1000/files/ixgbe%20stable/ http://sourceforge.net/projects/e1000/files/ixgbe%20stable/

• Ixgbevf driver version 2.16.1 -

http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/2.16.1/ http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/2.16.1/

After updating the driver, please retest using the above driver versions and provide the following in case the latest drivers doesn't address spoof

check issue.

• Host kernel and dmesg logs.

• Guest kernel and dmesg logs.

• Detailed setup instructions for reproducing the issue in house.

Hope the above helps.

rgds,

wb

[RBhat3]

Hi,

I am also having similar issue. I have downloaded latest ixgbe drivers and edited the source code to disable spoofing in functions "ixgbe_set_mac_anti_spoofing" and "ixgbe_set_vlan_anti_spoofing" and I have printed the values of the register(s) IXGBE_PFVFSPOOF(vfnum) and it showed a value of zero. I create one VF on each interface and assign that to a VM. On VM, have one application which sends VLAN tagged packets. For every packet, I get an error: "p7p1: 1 Spoofed packets detected". I tried disabling tx/rx VLAN offload and still see this error. Is there something I am missing?

regards,

Ravi

[SYeo3]

Hi Ravi,

Thank you for contacting Intel.

Were you able to try the command line in my post earlier? See below:

Since you are creating VF in the host. Once the VF's are created the Host OS loads the ixgbevf driver automatically.

Once the VF driver is loaded in the Host OS it will claim all the VF that it finds on the PCI bus.

These VFs are not allowed to be assigned to the VM at this point.

So, we believe this is the reason you are experiencing "Spoof Packet Detected" messages.

Please follow the procedure below:

1. Add "blacklist ixgbevf" to /etc/modprobe.d/blacklist.conf file.

2. Load ixgbe driver

3. Create VF using pci sysfs interface.

4. Assign VF to the VM

5. Boot VM

Please let us know your test results.

Sincerely,

Sandy

[RBhat3]

Sandy,

I tried blacklisting the ixgbevf yesterday itself. But of no success.

regards,

Ravi

[SYeo3]

Hi Ravi,

Thanks for your quick reply.

If you have updated the drivers as recommended by my colleague wb_intel, please send us the dmesg logs and your setup so we can further check on this.

Kindly refer to the recommendations below:

wb_Intel wrote:

Hi Shahamf,

Good day. As you are using ixgbe and ixgbevf drivers that are distributed inbox with Ubuntu. In box drivers are intended for basic connectivity and are maintained by the distribution publisher which is ubuntu in this case. Please try upgrading to latest ixgbe and ixgbevf drivers. Below are the driver version and download URL for the drivers.

• Ixgbe driver version 4.1.1 -

http://sourceforge.net/projects/e1000/files/ixgbe%20stable/ http://sourceforge.net/projects/e1000/files/ixgbe%20stable/

• Ixgbevf driver version 2.16.1 -

http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/2.16.1/ http://sourceforge.net/projects/e1000/files/ixgbevf%20stable/2.16.1/

After updating the driver, please retest using the above driver versions and provide the following in case the latest drivers doesn't address spoof

check issue.

• Host kernel and dmesg logs.

• Guest kernel and dmesg logs.

• Detailed setup instructions for reproducing the issue in house.

Hope the above helps.

rgds,

wb

We look forward to your reply.

Sincerely,

Sandy

[RBhat3]

Sandy,

I am using the latest driver from the sourceforge. Below are my steps on host side:

modprobe -r ixgbe

modprobe ixgbe max_vfs=1,1

ip link set p7p1 vf 0 mac 00:00:01:00:06:00

ip link set p7p2 vf 0 mac 00:00:01:00:06:10

ip link set p7p1 vf 0 vlan 100

ip link set p7p2 vf 0 vlan 200

ip link set p7p1 vf 0 spoofchk off

ip link set p7p2 vf 0 spoofchk off

ip link shows the below for p7p1:

10: p7p1: mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000

link/ether a0:36:9f:45:7d:40 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:00:01:00:06:00, vlan 100, spoof checking off, link-state auto

And on the guest, I had created a small raw ethernet socket based program and sending alternate normal and VLAN frames. (The sample program is attached). I see the normal ethernet packets going out with a VLAN tag added. for a VLAN packet it gives spoofed packet.

I even tried by changing the driver source code and forcing the registers to zero. I am using CentOS7 on host and guest.

I will try to get the kernel logs as it is taking time to copy them from remote machine.

regards,

Ravi