82599EB: hundreds of strange MAC addresses are seen on the switch

[ALyak]

Greeting all,

we are using 82599EB NICs on stock ubuntu natty, kernel 2.6.38-8 with ixgbe version 3.2.9-NAPI. Each NIC spawns SR-IOV VFs. Some of the VFs are attached to virtual machines with KVM PCI pass-through, and other VFs are used on the physical machine.

At some point, one of the PF-interfaces (eth103, specifically) started advertising hundreds of strange MAC addresses, which are seen on the switch. Here are some of them:

3 0a:00:00:00:00:00 te0/27dynamic 3 0a:00:00:00:10:11 te0/27dynamic 3 0a:00:00:01:85:ae te0/27dynamic 3 0a:00:00:29:38:39 te0/27dynamic 3 0a:00:00:77:3e:3e te0/27dynamic 3 0a:00:03:15:52:ef te0/27dynamic 3 0a:00:03:15:73:3d te0/27dynamic 3 0a:00:03:15:a7:a2 te0/27dynamic 3 0a:00:03:1f:32:34 te0/27dynamic 3 0a:00:03:1f:3d:34 te0/27dynamic 3 0a:00:03:1f:4a:9d te0/27dynamic 3 0a:00:03:1f:61:2d te0/27dynamic 3 0a:00:03:29:38:61 te0/27dynamic 3 0a:00:03:29:3e:3e te0/27dynamic 3 0a:00:03:29:52:ef te0/27dynamic 3 0a:00:03:29:61:2d te0/27dynamic 3 0a:00:03:29:73:3d te0/27dynamic 3 0a:00:03:29:a7:a2 te0/27dynamic 3 0a:00:03:33:00:00 te0/27dynamic 3 0a:00:03:33:1b:64 te0/27dynamic 3 0a:00:03:33:28:31 te0/27dynamic 3 0a:00:03:33:34:36 te0/27dynamic 3 0a:00:03:33:35:39 te0/27dynamic 3 0a:00:03:33:3d:34 te0/27dynamic 3 0a:00:03:33:3e:3e te0/27dynamic 3 0a:00:03:33:41:3d te0/27dynamic 3 0a:00:03:33:4a:9d te0/27dynamic 3 0a:00:03:33:52:ef te0/27dynamic 3 0a:00:03:33:61:2d te0/27dynamic 3 0a:00:03:33:73:3d te0/27dynamic 3 0a:00:03:33:a7:a2 te0/27dynamic 3 0a:00:03:33:aa:46 te0/27dynamic 3 0a:00:03:6c:3e:3e te0/27dynamic 3 0a:00:03:6c:61:2d te0/27dynamic

...

(there are hundreds more).

We don't have such MAC addresses in our environment. Some debugging with switch port monitoring and Wireshark suggests, that switch receives corrupted packets, parses them wrongly, and collects those non-existing MAC addresses. The result is that the interface in question is unable to successfully send any data to the switch port. (The reverse works, however: interface seems to successfully receive data from the switch).

Unfortunately, after some debugging the server was rebooted and the issue went away.

# Has anyone seen such phenomena or can suggest why it can happen?

# Is there a way to ask the 82599EB PF to dump its internal state, like what MAC addresses its internal L2 switch sees etc.? This can help in debugging future issues.

Thanks!

Alex.

[Patrick_K_Intel1]

Hi Alex,

I'll ask around. I've never seen this type of behavior, though my testing has been much more limited than what you are doing.

I am pretty sure the iproute2 utility, using the show option will show all VF's and the associated MAC addresses with them. That would be all that we have in our internal 82599 'switch'.

I assume eth103 was assigned to the kernel and not a VM.

Again, I'll ask around.

- Patrick

[ALyak]

Yes, Patrick, eth103 is the name of the PF on the physical server.

Thanks,

Alex.

[Patrick_K_Intel1]

Alex,

if you issue a "ip link show eth103" command, it will show all the VF's associated with that PF, and all of the hardware MAC addresses assigned to the PF. If other things are showing up, it is likely from outside the SR-IOV implementation.

If you have a VF assigned to a VM, could it somehow be trying to do some kind of MAC spoofing?

[ALyak]

Thanks, Patrick.

We are aware of the iproute2 utility, and were hoping for kind of internal "state dump" of the PF/VF driver. The VMs using the VFs are all under our control, and run exactly the same software. This issue showed up on a single PF, while the other PF (which is the second port of the same NIC) did not show such issue. Both PFs are used in a symmetrical way - both have VFs spawned and attached to same VMs.

If this issue shows up again, we will hopefully be able to debug more.

Thanks,

Alex.

[Patrick_K_Intel1]

Ethtool has an option to dump the registers of a device. You can use that and then look at any register within the Intel 82599 you wish. The datasheet that details the contents of the device is here: http://www.intel.com/content/www/us/en/ethernet-controllers/82599-10-gbe-controller-datasheet.html http://www.intel.com/content/www/us/en/ethernet-controllers/82599-10-gbe-controller-datasheet.html.

The MAC address table for the PF is detailed in the RAL/RAH registers – that is Receive Address LOW and Receive Address High and there are 128 of them. Then you can look in the MAC Pool Select Array MPSAR, which details which pool (VF) the MAC address is assigned to.

If you are able reproduce and investigate, would love tohear your results.