Rx drop for Ethernet 25G 2P E810-XXV

[Himanshu-Jain]

We have Dell R450 ubuntu machine with Intel(R) Ethernet 25G 2P E810-XXV.

Ubuntu version is 20.04.6 LTS and kernel version is 5.4.0-155-generic.

Ice driver version is 0.8.1-k and firmware version is 4.20 0x8001784f 0.0.0

Rx packet drop is seen when we are running a workload which consumes ~45-50% of the port.

Only 1 port NIC is connected as of now.

No Tx drop is seen.

In our workload load is more on Tx compared to Rx. Attaching SAR command output

$sar -n DEV 1 1 | grep eno12399

$ethtool -S eno12399 | grep drop

rx_dropped : 1703805100

tx_dropped_link_down_nic: 0

rx_dropped.nic :0

Above output suggest there is rx drop at driver or OS level not at NIC hardware level.

Issue is inconsistent, with both Rx & Tx stable for couple of hours then Rx drop seen for couple of hours.

We tried changing Rx Descriptor Ring size from 2048 to 8160 but that worsens the Rx drops.

$sudo ethtool -g en012399 - Attaching output of this command.

$sudo ethtool -G eno12399 rx 8160

Can you please help in debugging what can cause such issue.
Any commands for getting more information from driver?

Also not able to find documentation wrt driver version 0.8.1-k.
Is there way I can upgrade the driver?

Thanks

[B_Y]

Hi @Himanshu-Jain,

Thank you for posting in Intel Ethernet Communities.

You can download the latest Linux drivers from https://sourceforge.net/projects/e1000/files/ice%20stable/ 

1) May I know when the issue started? Any hardware or software changes happened?

2) May I have a confirmation if this Intel(R) Ethernet 25G 2P E810-XXV purchased separately or is it from OEM manufacturer together with the Dell PowerEdge R450?

You might need to contact Dell ( if your card is OEM NIC) for proper assistance to check the NVM and firmware drivers needed for the card.

This is the OEM / Dell driver based on your system model: Dell PowerEdge R450

https://www.dell.com/support/home/en-my/product-support/product/poweredge-r450/drivers

I'm looking forward to hearing from you.

Best regards,

BY_Intel

Intel Customer Support

[Himanshu-Jain]

Hi @B_Y ,

Thanks for your reply.

>>1) May I know when the issue started? Any hardware or software changes happened?

R450 Machine and NIC both are recently installed, lower work loads are working fine, facing Rx drop issue when we are moving towards higher workloads.

>>2) May I have a confirmation if this Intel(R) Ethernet 25G 2P E810-XXV purchased separately or is it from OEM manufacturer together with the Dell PowerEdge R450?

Both R450 and NIC are purchased from Dell together.

Thanks for sharing the link - https://www.dell.com/support/home/en-my/product-support/product/poweredge-r450/drivers

Couple of more observations after going through this link

1. Family firmware version currently installed is 22.0.9. Attaching screenshot for the same. As per above link this firmware version is for XL710 adapters - https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=9nppg&oscode=ubt20&productcode=poweredge-r450

So Is this a mistake by our installation team?

2. For E810 adapters, I am unable to get firmware version for Ubuntu - https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=25ffj&oscode=ubt20&productcode=poweredge-r450

 Is there a support for E810 adapters on Ubuntu?

Can you please share the instructions to update the ice driver. 

Any documentation available for ice driver  0.8.1-k and is it very old and is it recommended to move to newer versions?

Thanks

Himanshu

[B_Y]

Hi @Himanshu-Jain,

Thank you for the information.

1) Does increasing the RX ring buffer makes any difference, enter:

Example code:

# nmcli connection modify Example-Connection ethtool.ring-rx 4096

Based on your remark " XL710 adapters - https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=9nppg&oscode=ubt20&productcode=poweredge-r450" , theoretically speaking that is firmware for XL710 as per Dell description.

Can we ask your help to share with us in regards the photo of front and back markings of E810-XXV to validate?

2) Please refer to the link below for Supported Operating Systems for Retail Intel® E810 Ethernet Adapters:

https://www.intel.com/content/www/us/en/support/articles/000025890/ethernet-products.html

3) Please refer to the link below for Install Linux* Drivers for Ethernet Adapters:

https://www.intel.com/content/www/us/en/support/articles/000092128/ethernet-products/800-series-network-adapters-up-to-100gbe.html

As far as we are concerned, the ICE driver's release notes go back to 1.0.4:

- https://www.intel.com/content/www/us/en/download/19630/29872/intel-network-adapter-driver-for-e810-series-devices-under-linux.html

- https://sourceforge.net/projects/e1000/files/ice%20stable/1.0.4/

I'm looking forward to hearing from you.

Best regards,

BY_Intel

Intel Customer Support

[Himanshu-Jain]

Hi @B_Y ,

Thanks for your support.

>>1) Does increasing the RX ring buffer makes any difference, 

We tried changing Rx ring buffer to Maximum value of 8160 from 2048 by below command

$sudo ethtool -G eno12399 rx 8160

After this change Rx drop worsens and started occurring at more frequent intervals.

>>Based on your remark " XL710 adapters - https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=9nppg&oscode=ubt20&productcode=poweredge-r450" , theoretically speaking that is firmware for XL710 as per Dell description.

We confirmed from our team, The R450 shipped with the E810 NIC installed from the factory.  The 22.0.9 version is what the server shipped with.

Our team also mentioned Intel manual states the E810-XXV OCP is covered with 22.0.X. Attaching it for your reference.

What is the guidance from your side should we move away from what came from factory firmware version 22.0.9 to 21.5.0 family version - https://www.dell.com/support/home/en-in/drivers/driversdetails?driverid=25ffj&oscode=ubt20&productcode=poweredge-r450

>>Can we ask your help to share with us in regards the photo of front and back markings of E810-XXV to validate?
Sure, give us sometime. I don't have access to physical server as I am accessing remotely from India and server is in USA.

Thanks for sharing the links, I am going through them.

Not sure how we are getting ice driver version as 0.8.1-k

I was assuming this is default version with Ubuntu 20.04.6 LTS and kernel 5.4.0-155-generic.

Should we update just the driver or updater kernel so that driver will get updated automatically?

Thanks for your time

Himanshu

[B_Y]

Hi @Himanshu-Jain,

The usage of increase the RX ring buffer, may shortly interrupt the network connection. The max value of RX ring buffer is 8160. This value can be found out by "ethtool -g [interface]".

You may need to reload the NetworkManager connection, command:

# nmcli connection up Example-Connection

RX errors usually can be fixed with latest ice drivers, however since Dell R450 is a DELL server that came together with the E810-XXV.

From this thread, a developer found an issue regarding random packet loss from old driver. Please refer if it is able to help out.

We wish to suggest to update the latest driver provided by DELL first since this is an OEM, you may need to reach out Dell Support for accurate driver.

The list of latest drivers based on Dell R450 are the generics, it would appear different if you use with a Dell Service Tag (The Service Tag and Express Service Code are found on the front of the system by pulling out the information tag. Alternatively, the information might be on a sticker on the back of the chassis. The Dell EMC service tag is a five or more character identifier that is unique to your product.)

Thank you.

Best regards,

BY_Intel

Intel Customer Support

[B_Y]

Hi @Himanshu-Jain,

I hope this message finds you well. I am just sending a follow up if you have any questions from the previous post. If so, please let us know. Thank you.

Best regards,

BY_Intel

Intel Customer Support

[Himanshu-Jain]

Hi @B_Y 

Thanks for your help and guidance.

Increasing Rx buffer didn't helped us.

Finally we ended up installing new HWE kernel 5.15, we were on 5.4 before and it fixed RX drop issue.

I am attaching output from new kernel for ice driver.

I have one question like how to determine ice driver version, I am getting version like 5.15.0-79-generic. But as per below link all versions are of 1.x.y format - https://sourceforge.net/projects/e1000/files/ice%20stable/ 

Thanks & Regards

Himanshu

[B_Y]

Hi @Himanshu-Jain,

Glad that the issue has been resolved by updating to Ubuntu version 20.04.6's latest kernel. Before answering your last query, may I ask for your assistance with this command to get its output:  Ethtool -i 

Thank you.

Best regards,

BY_Intel

Intel Customer Support

[Himanshu-Jain]

Hi @B_Y 

Please find ethtool output attached.

Thanks

Himanshu

[B_Y]

Hi @Himanshu-Jain,

I appreciate you re-attaching the ethtool -i output. Please give us some time so we can look into your inquiry.

Best regards,

BY_Intel

Intel Customer Support

[Himanshu-Jain]

Hi @B_Y 

Thanks, will wait for your reply.

Regards

Himanshu

[B_Y]

Hi @Himanshu-Jain,

Thank you for your patience.

The summary of this scenario:

Before applied with latest kernel, the output was 0.8.1-k.

Actual Issue has resolved with Ubuntu version 20.04.6's latest kernel (5.15), the output is showing version like 5.15.0-79-generic

The reason is most likely because it was updated with latest kernel and starts showing the kernel version.

These are the methods to determine driver version:

modinfo ice

sudo lshw -class network

You may also refer to the ice-1.0.4.tar.gz release note for further details, from:

Intel® Network Adapter Driver for E810 Series Devices under Linux*

Please let me know if i can be of further assistance.

Best regards,

BY_Intel

Intel Customer Support

[B_Y]

Hi @Himanshu-Jain,

This is just a follow up regarding the information we have shared in regards your inquiry. If you have additional questions or clarifications, feel free to let us know.

Best regards,

BY_Intel

Intel Customer Support

[Himanshu-Jain]

Hi @B_Y ,

Commands like ethtool, modinfo and lshw all are giving ice driver as kernel version "5.15.0-79-generic". Attaching output of the same.

Is there any way to get know which version of ice driver 1.x.y is built as part of kernel, then I can update the ice driver.

Regards

Himanshu

[B_Y]

Hi @Himanshu-Jain,

Please allow 1-3 business day for us to check internally and respond accordingly. Thank you.

Best regards,

BY_Intel

Intel Customer Support

[B_Y]

Hi @Himanshu-Jain,

Thank you for your patience. 

You may try with # lsmod

- Using "lsmod" command to display a list of all of the modules that are used by the Linux Kernel.

- Make a note of the module for which you will need to verify the driver version that is installed.

Note: If you did manually build the driver, such as:

1. Move the base driver tar file to the directory of your choice.

For example, use /home/username/ice or /usr/local/src/ice.

2. Untar/unzip the archive, where <x.x.x> is the version number for the driver tar file:

# tar zxf ice-<x.x.x>.tar.gz

3. Change to the driver src directory, where <x.x.x> is the version number for the driver tar:

# cd ice-<x.x.x>/src/

4. Compile the driver module:

# make install

- modinfo [driver-name] should tell that driver module that you have compiled.

The conclusion: The installed ice driver most likely from the kernel update: 5.15 and that may be a the reason it is showing the kernel version as the current. Meanwhile, where ice-1.12.7.tar.gz, <x.x.x> is the version number for the driver tar file.

The binary will be installed as:
/lib/modules/<KERNEL VER>/updates/drivers/net/ethernet/intel/ice/ice.ko

I hope this helps with your inquiries.

Best regards,

BY_Intel

Intel Customer Support

[B_Y]

Hi @Himanshu-Jain,

I hope this message finds you well. I am just sending a follow up if you have any further inquiry. If so, please let us know. Thank you.

Best regards,

BY_Intel

Intel Customer Support

[B_Y]

Hi @Himanshu-Jain,

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

Best regards,

BY_Intel

Intel Customer Support