X540-T2 VF cannot receive packet with OVS

[RTrac1]

I have configured host with two PF and 16 x2 VF in debian system. It worked well if I use the PF/VF as standalone NIC, but not functional with OVS bridge.

The testing scenario as below:

eth0 (PF): 10.250.11.103/24

eth11 (VF 9): 10.250.11.113/24

Bridge "vmbr1"

Port "eth10"

Interface "eth10" (VF 8)

Port "vmbr1"

Interface "vmbr1"

type: internal

Port "in1"

Interface "in1"

type: internal

ovs_version: "2.6.0"

in1: 10.250.11.34/24

I try the following icmp test:

ping 10.250.11.254 (reply good)

ping 10.250.11.254 -I eth11 (reply good)

ping 10.250.11.254 -I in1 (NO response)

ip link show my VF as:

-------------------------------

2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 0c:c4:7a:df:36:c6 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 6e:74:9e:65:58:eb, spoof checking on, link-state auto

vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 8 MAC da:c7:06:0b:5b:1f, spoof checking on, link-state auto

vf 9 MAC ea:97:12:4e:b4:6b, spoof checking on, link-state auto

vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

arp table shown as:

-------------------------------

Address HWtype HWaddress Flags Mask Iface

10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth11

10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth0

10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C in1

But arp is not stable, it disappear very quickly:

------------------------------------------------------------

Address HWtype HWaddress Flags Mask Iface

10.250.11.254 (incomplete) in1

10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth0

10.250.11.254 ether 6c:3b:6b:e9:7c:e1 C eth11

The strange thing is:

ping via in1 will not fill up the arp table for in1 Iface (always show incomplete), but ping via other two iface seems fill the mac address into in1 some time.

I have tcpdump to in1 and found it send out ARP request broadcast, but did not get any ARP reply from host 254.

ovs-appctl fdb/show vmbr1:

-----------------------------------------

port VLAN MAC Age

2 0 72:fc:b4:3a:e6:0f 78

1 0 6c:3b:6b:e9:7c:e1 38

1 0 6c:3b:6b:f5:96:e6 3

1 0 88:43:e1:dd:9c:80 1

This looks persist and will not change whatever I ping via any Iface.

I have also try to change VF port, but seems none of them work with OVS bridge.

But the bridge worked if assigned the port to eth0 or eth1 PF.

uname -a:

Linux pve3 4.4.62-1-pve # 1 SMP PVE 4.4.62-88 (Thu, 18 May 2017 09:18:43 +0200) x86_64 GNU/Linux

/etc/network/interfaces:

----------------------------------

auto eth0

iface eth0 inet static

address 10.250.11.103

netmask 255.255.255.0

gateway 10.250.11.254

auto eth11

iface eth11 inet static

address 10.250.11.113

netmask 255.255.255.0

allow-vmbr1 eth10

iface eth10 inet manual

ovs_type OVSPort

ovs_bridge vmbr1

auto vmbr1

iface vmbr1 inet manual

ovs_type OVSBridge

ovs_ports eth10 in1

allow-vmbr1 in1

iface in1 inet static

address 10.250.11.34

netmask 255.255.255.0

ovs_type OVSIntPort

ovs_bridge vmbr1

Here is ethtool info for eth10:

-------------------------------

# ethtool -i eth10

driver: ixgbevf

version: 2.12.1-k

firmware-version:

bus-info: 0000:05:12.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: no

supports-register-dump: yes

supports-priv-flags: no

# ethtool -k eth10

Features for eth10:

rx-checksumming: on

tx-checksumming: on

tx-checksum-ipv4: off [fixed]

tx-checksum-ip-generic: on

tx-checksum-ipv6: off [fixed]

tx-checksum-fcoe-crc: off [fixed]

tx-checksum-sctp: on

scatter-gather: on

tx-scatter-gather: on

tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: on

tx-tcp-segmentation: on

tx-tcp-ecn-segmentation: off [fixed]

tx-tcp6-segmentation: on

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on [fixed]

tx-vlan-offload: on [fixed]

ntuple-filters: off [fixed]

receive-hashing: off [fixed]

highdma: on [fixed]

rx-vlan-filter: on [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

tx-gre-segmentation: off [fixed]

tx-ipip-segmentation: off [fixed]

tx-sit-...

[idata]

Hi Ray,

Thank you for posting in Wired Ethernet Community.We're currently checking your issue and will update this thread as soon as possible. regards,Vince

[idata]

Hi Ray, while we're still checking your issue, please try the latest ixgbe VF version 4.1.2. kindly refer to download link below.

https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/4.1.2/regards,Vince

[idata]

Hi Ray, we'd like to follow-up if you've tried the ixgbe VF version 4.1.2. thanks.

regards,Vince

[RTrac1]

I am sorry, the infra team has changed the design, and I have no environment can be tested with new version.

I would like close the case first.

[idata]

Hi Ray, thanks for sharing the update. Feel free to create a new thread if you have any question related to Intel® Wired Ethernet Network Adapters.

regards,Vince