[ Intel E810-XXVDA2] tx_error on linux when LLDP encapsulated in qinq frame

[ACarpi]

Consider the capture attached, containing LLDP within a Q-IN-Q frame generated by one of our Ubuntu servers.  

How to reproduce
==============
We'll be trying to replay the capture containg LLDP inside qinq frames attached in this report. 

1. On one console, watch for tx_errors before with the following command
watch -n0 --diff 'sudo ethtool -S dutuif0 | grep tx_errors'

2. On a second console, replay the packet capture using tcpreplay
sudo tcpreplay -i dutuif0 -t -K --loop 600  tx_errors.pcap

This capture will increment the number of tx_errors by exactly 4200. There are 7 of such LLDP+qinq frames in the capture. Any other frame (~1880 others) are sent just fine.


About the setup

=============

The NIC card is installed on a Dell C6620 running Ubuntu 22.04. Below the details of our OS, Kernel and driver versions. Firware LLDP agents are disabled, as you can see from the output of ethtool in the last command. So the LLDP handling is done entirely in software by the OS

apstrktr@bs603-03r:~$ uname -a
Linux bs603-03r.dc1.apstra.com 5.15.0-105-generic #115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

apstrktr@bs603-03r:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"

apstrktr@bs603-03r:~$ sudo ethtool -i dutuif0
driver: ice
version: 5.15.0-105-generic
firmware-version: 4.40 0x8001ba1e 22.5.7
expansion-rom-version:
bus-info: 0000:17:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

apstrktr@bs603-03r:~$ ethtool --show-priv-flags dutuif0
Private flags for dutuif0:
link-down-on-close     : off
fw-lldp-agent          : off
vf-true-promisc-support: off
mdd-auto-reset-vf      : off
legacy-rx              : off

And I also tried this newer version of the ice driver, downloaded from here, same result.

apstrktr@bs603-02l:~$ sudo ethtool -i dutuif0
driver: ice
version: 1.13.7
firmware-version: 4.40 0x8001ba1e 22.5.7

 An additional piece of information: the issue looks similar to the one discussed here  with a caveat: the check on "ethhdr->h_proto" in this case doesn't match  ETH_P_LLDP (0x88CC) because the LLDP frame is inside qinq (so ethertype matches 0x88A8 instead), and the "real" type to check is 2 bytes ahead. I believe the check to send out LLDP should include this condition as well. 

Of course the same identical traffic pattern is handled correctly with any of the previous intel NIC models we have internally, all using the "ixgbe" driver (e.g. x710-da2 x520-da2). 

There are several reasons why we generate traffic in such a way, primarily to create virtual topologies running several virtual NOSes (Juniper vQFX, Cisco NXOSv, Arista vEOS etc) that are deployed among different KVM servers. The outer tags are generated/terminated by openvswitch, and the underlaying switching infrastructure is configured to forward these BPDUs  transparently as any other frame, so that pseudo-wires connection can be created among VMs deployed on different servers.

[Azeem_Intel]

Hi ACarpi,

Greetings for the day!

Hope you are doing well. Could you please share a snapshot of the defective part with the serial number visible?. Additionally also let us know if this product was shipped with the system or if you bought it separately?

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi ACarpi,

Greetings for the day!

This is the first follow-up regarding the issue ([ Intel E810-XXVDA2] tx_error on linux when LLDP encapsulated in qinq frame) you reported to us.

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided.

Best Regards,

Azeem_Intel

[Azeem_Intel]

Hi ACarpi,

Greetings for the day!

This is the second follow-up regarding the reported issue. We're eager to ensure a swift resolution and would appreciate any updates or additional information you can provide.

Best Regards,

Azeem_Intel

[ACarpi]

Hi Azeem,

I wanted to make sure it's clear that this is not due to a potentially defective product, but it has been verified on a set of systems all with identical configurations (20 nodes all showing the same behavior) and RCA'd to be a clear linux driver issue. 

This card is part of a Dell C6620 build (so it's a Dell certified configuration) I can certainly attach a picture of one of the card (attached here), but not sure it will be really helpful, as again this is not a "defective" product but a bug in the driver.

Since this is a blocking issue for us, all these Intel NICs have been currently swapped. I highly recommend you have your linux driver team look at this issue and reproduce referring to my first message: it it's a single tcpreplay command using the capture I provided, and they will certainly be able to confirm the issue.

I will not be able to work on this problemn  as I no longer have the cards, but before moving forward I wanted to open this support ticket to give you an opportunity to understand and fix the issue in future versions of the driver. Again, this should be really easily reproducible on your setup.

Best regards,

[vij1]

Hello Acarpi,

Greetings!

Thank you for the response. Once we got an update we will keep you posted.

Regards,

Vijay N.

[Azeem_Intel]

Hi ACarpi,

Greetings for the day!

Thank you for the update. Since the card was shipped with the Dell C6620 system, which is an OEM machine, we recommend reaching out to the OEM team for better assistance with your inquiry. They will be in a better position to provide the necessary support for the integrated components of your system.

Best Regards,

Azeem_Intel

[ACarpi]

Azeem,

I would prefer being redirected to a mailing list or issue tracker for the linux ice driver. Can you please provide information of what is the best place to create issues regarding the linux driver for this product, so I can get in touch with the driver development team?

Best regards,

Alessandro

[Azeem_Intel]

Hi ACarpi,

Greetings for the day!

Thank you for your prompt response. As part of the server hardware support team, we may not have specific information on Linux compatibility. However, I recommend checking the Linux website for any compatibility information regarding your inquiry. They may have relevant resources or community forums that could assist you further. Additionally, since this system is OEM (Dell), I would suggest contacting the OEM team directly for more tailored assistance., Thank you.

Best Regards,

Azeem_Intel