i40e driver: X710 Firmware returns error-code 0x0005

[NChau13]

Hi,
we see following very often:

2020-10-19T21:28:20.160+09:00 TYO-PL9420 kernel: i40e 00.0 AQTX: Command completed with error 0x5.
2020-10-19T21:28:20.160+09:00 TYO-PL9420 kernel: i40e 00.0 AQTX: desc and buffer writeback:
2020-10-19T21:28:20.160+09:00 TYO-PL9420 kernel: i40e 00.0 AQ CMD: opcode 0x0628, flags 0x2007, datalen 0x0000, retval 0x0005
2020-10-19T21:28:20.160+09:00 TYO-PL9420 kernel: i40e 00.0      cookie (h,l) 0x00000000 0x00000000
2020-10-19T21:28:20.160+09:00 TYO-PL9420 kernel: i40e 00.0      param (0,1)  0x00000000 0x00000000
2020-10-19T21:28:20.160+09:00 TYO-PL9420 kernel: i40e 00.0      addr (h,l)   0x00000000 0x00000000
2020-10-19T21:28:20.160+09:00 TYO-PL9420 kernel: i40e 0000:08:00.0: Failed opcode 1028 Error: -53

 As it can be seen, the PF returns the code (-53)  i.e. I40E_ERR_ADMIN_QUEUE_ERROR which I see getting returned from Linux kernel driver.

Based on Linux kernel code browsing, I see that when the Firmware returns 0x0005 as shown in above snippet, the Linux i40e PF driver treats it like an error and returns the  I40E_ERR_ADMIN_QUEUE_ERROR to VF.

We couldn't understand when this actually happens.

It gets resolved after sometime, say couple of seconds.

But, for those two seconds, retrying does not help.

Please help us understand the reason behind it - why the NIC firmware returns the 0x0005 error-code as can be seen from the above log snippet which is enabled via ethtool.

We tried on multiple different NICs, and this issue is seen on all of them eventually like once in 20 to 200 consecutive iterations of a test.

Thanks,
- Nachiket C.

[AlfredoS_Intel]

Hi Nchau13,

Thank you for posting in our Intel® Ethernet Communities Page.

We are sorry to hear about the issue that you are experiencing with your network adapter.

So we would have a better understanding of your issue, please provide the following information:

1. Please download and run our Intel® System Support Utility from this page, https://downloadcenter.intel.com/download/26735/Intel-System-Support-Utility-for-the-Linux-Operating-System#:~:text=Intel%20SSU%20for%20the%20Linux,and%20shared%20by%20the%20user. After running it, you will be given an option to save the logs to a text file, please do so and attach the file on your reply.

2. Kindly provide us the results of this command: ethtool -i ethx where ethx is the Ethernet port.

We look forward to hearing from you. If we do not get your reply, we will follow up after 3 business days.

Best Regards,

Alfred S

Intel® Customer Support

[AlfredoS_Intel]

Hi Nchau13,

We are just following up.

It looks like you need more time to provide the information that we asked.

We will follow up again after 3 business days. Should we not hear from you, our system may automatically close the thread.

Best Regards,

Alfred S

Intel Customer Support 

[NChau13]

Hi @AlfredoS_Intel 

I am sorry for late response.
Initially, I couldn't get the update on ticket so we raised a ticket on the IntelPremierService platform - https://premiersupport.intel.com/IPS/5003b00001XtcXs

I think it will be better if we track this issue at one place alone.

Thanks a lot for your interest to help us out.

- Nachiket C.

[AlfredoS_Intel]

Hi Nchau13, 

Thank you for your update.

It is a great joy to know that you are already in a path to resolution for your issue.

If you would like to give us an update or if you have further questions, please submit a new question as this thread will no longer being monitored.

Thank you for contacting Intel® and have a great week!

Best Regards,

Alfred S

Intel® Customer Support