ARP request duplication on XL710

[JGutt]

Hi,

I'm seeing something really disturbing on our setup, which causes severe problems for us:

If you send a broadcast ARP request out using tcpreplay on the interface, the packet gets reflected back. To observe this, it's merely necessary to tcpdump on the interface that you are sending the packet out on: the link partner will receive one packet, but you will sniff two: the transmitted packet, and one that got somehow reflected back.

I've tested with another XL710 link partner and another third party link partner: this seems to happen internally on the XL710 on egress.

# tcpdump -n -i p1p1

tcpdump: WARNING: p1p1: no IPv4 address assigned

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes

17:47:47.135804 ARP, Request who-has 192.168.47.5 tell 192.168.47.1, length 28 <---- this one is outbound

17:47:47.135861 ARP, Request who-has 192.168.47.5 tell 192.168.47.1, length 46 <---- this one is inbound

We're using Ubuntu 14.04, with the following driver and firmware versions:

driver: i40e

version: 1.2.37

firmware-version: f4.33.31377 a1.2 n4.42 e1932

lspci says:

01:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller LX710 for 40GbE QSFP+ [8086:1583] (rev 01)

Is there any way that this behaviour can be disabled? It causes severe problems that can escalate into packet storms for us.

[GREGORY_R_Intel · Accepted solution]

Greetings,

I'm sorry to hear you're having problems with duplicated packets on the XL710 adapter. I think the problem here is that the tcpdump program will normally request that the capture device enter "promiscuous mode". However, the XL710 semantics for promiscuous mode have changed from previous Intel products. In our previous products promiscuous mode only referred to the physical interface so that packets appearing on the physical port would be captured no matter what their MAC address was. The XL710 has additional support for VEB bridge configuration and promiscuous mode will now also apply to any Virtual Station Interface (VSI) such that any packet sent from any VSI, including the main VSI, will also be replicated and copied back.

There are two ways to avoid this behavior. One is to use a filtering utility to add the source MAC address of the ARP packet to the main VSI filters. In Linux you can use the bridge utility to do this.

'bridge fdb add ...'

Another alternative is to use not use promiscuous mode on tcpdump. You can turn off promiscuous mode on the tcpdump utility by specifying the '-p' option.

I hope this helps resolve your issue. If you have further questions do not hesitate to contact us for more help.

Regards,

- Greg Rose

[JGutt]

So, I presume this is intended behavior? This means that implementing a simple learning bridge would not be possible with the XL710. Both workarounds would break a normal learning switch.

I guess the confusion lies with the standard assumption that a bridge generally floods frames to all ports except the port it received the frame on.

Is there any way to force a direct mapping between the VSI and the physical port?

Thanks,

- Jan Gutter

[GREGORY_R_Intel]

No, this is not intended behavior - it is a problem with source pruning. It will be fixed in future products but for now we are stuck with the "feature" on the XL710. The part could be used in a learning bridge scenario as long as a SW work around for the source pruning issue were implemented. Using the 'bridge fdb add' command is essentially a manual work around for the problem.

With the SW workaround in place then the mapping between the main VSI and the physical port is direct in most cases I can think of but I can't say for sure that I've seen all corner cases, etc.

Regards,

- Greg

[JGutt]

Thanks very much for the info: we'll see what we can implement as a workaround on the CPU side.

Any possibility that this could be fixed in a future firmware revision?

- Jan Gutter

[GREGORY_R_Intel]

There is no planned FW revision or update to address this issue - the Linux driver README does document the workaround as described above.

- Greg

[Jesse_B_Intel]

We have a patch making its way upstream and will be in our next release to work around this issue (as long as you're not trying to use SR-IOV, or NPAR modes on the adapter)

https://patchwork.ozlabs.org/patch/470209/ https://patchwork.ozlabs.org/patch/470209/