Drop tcp traffic x710 network port

[harshadineth]

I have intel X710 ethernet controller and recently I updated the Kernel to "6.5.0-21-generic" version from "5.15.0-97-generic" and after upgrading I have seen packet drops and I got tcp connection timeout from application logs. 

When I use the "ethtool -S enp25s0f0 | grep drop" command i can see the dorps are increasing. So Ring buffer value is 512 and I changed it to 256, 1024, 2048, 4096 but still i am seeing the drops. Then I rolled back to the old kernel version and then the issue resolved. Below are the driver and firmware version details. 

driver: i40e
version: 6.5.0-21-generic
firmware-version: 8.50 0x8000be1e 1.3082.0
expansion-rom-version:
bus-info: 0000:19:00.1

I am not seeing this issue in my I350 Gigabit Network Connection NIC cards. 

[Sazirah]

Hi harshadineth,

Thank you for posting in Intel Ethernet Communities.

Regarding your issue, you may want to refer to below link on how to resolve this issue:

https://www.intel.com/content/www/us/en/support/articles/000058714/ethernet-products/700-series-network-adapters-up-to-40gbe.html

Let us know the outcome once you performed the troubleshooting steps.

Regards,

Sazzy_Intel

[ESNewmanium]

Hi @harshadineth,

I saw the same thing across several servers starting in 6.5.0-15-generic in Ubuntu. We suspect that this patch may be relevant in fixing the issue: https://patchwork.kernel.org/project/netdevbpf/patch/20231004083454.20143-1-tirthendu.sarkar@intel.com/

Try 6.5.0-25 and see if the issue happens again. https://launchpad.net/ubuntu/+source/linux/6.5.0-25.25

For us at least, this packet loss issue on i40e only affected TCP streams and had no effect on ICMP (ICMP was perfect 0% packet loss while TCP clients were dropping repeatedly).

[harshadineth]

Thanks @ESNewmanium i will apply this patch and check the status. We also didn't observe any packet loss for ICMP traffic. Only TCP traffic is impacted. 

[garbinc]

Did this work for you? I’ve been having issues with kernel 6.5.x

I reverted to Debian on 6.1 and all is good.

[Sazirah]

Hi harshadineth,

Let us know the outcome once you have complete the troubleshooting steps.

Regards,

Sazzy_Intel

[harshadineth]

Hi Sazzy_Intel,

I have installed the latest driver and firmware in one of my server but unfortunately it doesn't solve the issue. I think, working without issues with old kernel version will rule out all hardware issues ( PCI, SFP, Cables etc )

Also I changed the ring buffer size to different values but even that doesn't help.

[vij1]

Hi Harshadineth,

Greetings for the day!

Thank you for contacting us. We want to assure you that we are actively working to find an appropriate resolution for your issue. Rest assured, we will provide you with an update as soon as possible. Your cooperation during this time is greatly appreciated.

Please feel free to reach out to us if you need any further assistance.

Regards,

Vij

[Sazirah]

Hi harshadineth,

We are currently checking on the resolution and will get back to you.

Regards,

Sazzy

[harshadineth]

Hi Sazzy,

Any update on this please. 

Regards

Harsha

[Sazirah]

Hi harshadineth,

Good day.

Regarding your issue, please refer to X710 Feature support Matrix for correct versions to use. Intel® Ethernet Controller X710/XXV710/XL710 Feature Support Matrix

 Could we know if you are using the in-box driver or out-of-tree driver? It appears you are using out-of-box. You should use out-of-tree. For an 8.5 NVM that would be Linux driver 2.17.4

From the X710 Feature support matrix - supported Linux Stable Kernel versions are 5.13/5.10/5.7/5.4/4.x/2.6

Regards,

Sazzy_Intel

[Sazirah]

Hi harshadineth,

We would like to know if the provided resolution was able to help in your situation. In case you did, we would highly appreciate if you would let us know the outcome of these. We will be waiting for your response.

Regards,

Sazzy_Intel

[Sazirah]

Hi harshadineth,

Good day

This is regarding your enquiries above. Since we have not seen an update, if you need any additional information, please submit a new question as this thread will no longer be monitored.

Best Regards,

Sazzy_Intel

[ESNewmanium]

I'm not the original poster here but I can confirm that moving up to 6.5.0-32 fixed the issue. I was able to reliably reproduce the issue on 6.5.0-15 and it's not happening at all on 6.5.0-32. The i40e patch "sync next_to_clean and next_to_process for programming status desc" does seem to fix it.