i40e NVM 8.40 breaks qinq

[Ben-TC]

We believe we have identified a possible bug in NVM 8.40 for i40e.  When running 8.15 VLAN and QinQ stripping both work but when upgrading to 8.40 only the inner vlan is stripped.

The full writeup and some additional testing has been posted to the DPDK mailing list.  Posting here as well in case someone can verify/tell us what we are missing.

* https://mails.dpdk.org/archives/dev/2022-April/238185.html

* https://mails.dpdk.org/archives/dev/2022-April/238269.html

Thanks,

Ben

[Zigfreid_Intel]

Hello Ben-TC,

Thank you for posting on Intel Ethernet Communities. 

We'd like to set your expectation that for Original Equipment Manufacturer(OEM) Ethernet adapter that you have which is an "i40e XL710", we have limited support and it would be best if you can also get in touch with the manufacturer for further support since they have customized drivers, firmware and software to enable or alter features on your card. They are in best position where you can seek assistance so you don’t lose features or customizations of the Ethernet adapter. But don't worry as we will still provide our best effort support to help you resolve the issue. 

Please provide us some to have this check, is it okay if we can ask you to provide us between 2-3 business days for this? If you have a preferred date please advise us instead. Thank you and have a nice day!

Best regards,

Zigfreid I.

Intel® Customer Support

[Zigfreid_Intel]

Hello Ben-TC,

We thank you for providing us the needed information on your concern however, as per further checking as much as we want to assist you our support is limited since you are using a 3rd party Ethernet adapter. We understand that this is an Ethernet controller that's embedded with Dell adapter. For 3rd party Ethernet adapter, it would still be best for you to get in touch with Dell for further support since they have customized firmware to enable or alter features on your adapter. They are in best position where you can seek assistance so you would not lose features or customizations of the Ethernet adapter.

If you have questions or other clarifications please let us know so we can help you further

Awaiting to hear from you soon, should there be no response from you, I’ll make sure to reach out after 3 business days. Thank you and have a nice day!

Best regards,

Zigfreid I.

Intel® Customer Support

[Shaq_Intel]

Hello Benjamin,

Sorry for the late update regarding your issue, since the issue still under investigation between engineering and dev team. This is the latest update from engineering team. 

"After further testing with the DPDK engineering team, we have found that when both strip options are enabled with 9.10 NVM, the behavior is the same as 8.15 when only one stripping type is enabled. We have shared this finding with the dev team and will discuss it more."

I will update you again once there is any new update regarding this. 

Best regards,

Shaq_Intel

[Shaq_Intel · Accepted solution]

Hi Benjamin,

Our Engineer and Dev believe they have found the issue. In testing, they see the problem behavior when VLAN settings are configured via the command-line at testpmd start time. If VLAN settings are configured during testpmd runtime, then inner and outer stripping works as expected and as seen with the 8.15 NVM. 

They have compared the packet captures from the following test done with 9.10 NVM, with captures from the previous method on 8.15 NVM, and see identical results.

Can you try to test with the following commands to see if this resolves the issue.

Start testpmd without any VLAN configuration:

./dpdk-testpmd -c 0xffff -- -I

After testpmd has started, set VLAN configuration:

vlan set extend on port_num

vlan set filter on port_num

vlan set strip on port_num

vlan set qinq_strip on port_num

Shaq_Intel

[Shaq_Intel]

Hi Benjamin,

Good day and hope that you are doing well.

This is just a follow up regarding the issue of your reported with us. We know that this is important for you to get it resolved and it is also equally important for us to get you the right solution.

Since we have not seen an update for several days, hence we will proceed with the case closure after 2 business day.

You can still re-open the case by logging into your web portal or replying to this email.

Please note that if the case is re-opened more than 15 days after closure, a new case will be created with a reference to the original case. 

Shaq_Intel

[Ben-TC]

Hello,

We had a chance to load the new firmware and run a very quick test but do not believe this is working correctly yet with 9.10.  When downloading the firmware we noticed that 9.20 was alsos available.  Did they mean to be using 9.20?  We need to complete further testing.  Quick testing shows that it does strip an outer vlan but does not strip both vlans.  We added patches to testpmd (https://patches.dpdk.org/project/dpdk/patch/20220404005634.2657-1-koncept1@gmail.com/) to show both inner and outer vlans, from testing only one vlan as observed in the logs.  Specifically it lacked all of the expected offload flags showing multiple vlans were stripped.

[Shaq_Intel]

Hi Ben

Have you try the command that our dev provide and it still having same issue? I believe they mention 9.10 nvm.

Shaq_Intel

[Ben-TC]

Confirming this is still not working as expect in 9.10 following the steps provided.  We have also reached the point where we believe we need to write this off as unlikely to be fixed.  It will push us to likely develop a solution on our own in software...

While the traffic may be stripped, we loose the stripped vlan information which we need access to.  See attached outputs from testpmd under both 8.15 and 9.10.

I am hesitant to try 9.20 since I don't see a downgrade package available.  There is a 9.01 downgrade that is listed as the latest but that version doesn't match 9.20 like the others match.  We don't want to be stuck with one card on an inconsistent firmware version.  There is a 9.10 downgrade package which we used to perform the 8.15 test attached here.

[Shaq_Intel]

Hi Benjamin,

Thank you for the update and the data that you provide, I will forward it to our Engineering team for them to analyze it further.

I will update you again if there is any finding from them.

Regards,

Shaq_Intel

[Shaq_Intel]

Hi Ben-TC,

While our engineering department still analyze the data that you provide. They also mention that "Regarding downgrades from NVM 9.20. Please assure the customer that there will be downgrade files available this week for 9.20." Maybe you can try to update to the NVM 9.20 to see if it fixed your issue. Since there will be a downgrade from 9.2c

Shaq_Intel

[Shaq_Intel]

Hi Ben-TC

As this case will take longer for the Engineering Team to analyze. I will temporary close this case and will send you an email if there is any update regarding this. As for this thread will be closed and not monitored from now.

Shaq_Intel

[Irwan_Intel]

Hi Ben-TC

We hope this message finds you well. We wanted to provide you with an update regarding the issue we discussed in our previous communication.

Upon further examination, we have discovered that the patch we mentioned is unrelated to the issue you reported. After further discussion, this is believed to be working as expected. Because we are seeing that the stripped packet reports an ether type of 0x800, this indicates to us that stripping is working as expected, and the tags have already been stripped by HW.

This can be seen in your uploaded text files comparing the 8.15 and 9.10 FW results. Both show the type of the received packet to be 0x0800.

We appreciate your patience and understanding throughout this process.

If you have any further concerns or questions, please do not hesitate to contact us.

Regards,

Irwan_Intel

[Ben-TC]

As stated previously we've given up on this ever being fixed and continue to dispute your "working as designed".  The stripping may be occurring correctly but there is information lost in the process.  Please note that in 9.10 there is no record of the outer vlan that was stripped unlike the 8.15 which produces a log message with both vlans in it.  This is a problem for us as we need to know both the outer and inner vlans.