X550-T2 NIC on Ubuntu 20.04: link down but still seen up on the switch side

jfr · ‎06-07-2021

Hello,

we have HP CL2600 servers with X550-T2 NICs, running on Ubuntu 20.04. The NICs are connected to Cumulus AS5812 switches.

# lspci | grep X550
17:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
17:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)

When we set an interface down on Linux (with ip link set down ens1f0 for instance), the link is still shown as up on the Cumulus switch side. It seems that even if you bring the interface down on Linux, the link is still physically up.

Cumulus support told us that there is a way to set a parameter Keep_PHY_Link_up_En that would fix this behavior. What is the right way to do it on Linux?

# ethtool -i ens1f0 
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x8000156d, 1.2926.0
expansion-rom-version: 
bus-info: 0000:17:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Let me know if you need more information.

Thanks,

Julien

Caguicla_Intel · ‎06-08-2021

Hello Julien,

Thank you for posting in Intel Ethernet Communities.

Can you please share more information on your intended configuration/setup for this query and why do you need to set the interface down?

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support

jfr · ‎06-08-2021

Hello Crisselle,

our network interface is dual-port, each port is part of a bond and connected to a different Cumulus switch. I was doing some failover tests by shutting down a port of the interface on my Ubuntu server to check that the bond works correctly and that we do not lose network connectivity.

The problem is that the switch still sees the down interface as up on its side, which is not the correct behavior IMHO.

I have read in the user manual that it is possible on Windows to disable the link if the interface is brought down. Is it possible to configure the same behavior on Linux?

Regards,

Julien

Caguicla_Intel · ‎06-08-2021

Hello Julien,

Thank you for the swift response.

Can you please share the link where the attached screenshot was taken?

Please also confirm if you already tried the latest driver from link below then set an interface down using "ip link set dev <interface> down"?

Intel® Network Adapter Driver for PCIe* Intel® 10 Gigabit Ethernet Network Connections under Linux*

https://downloadcenter.intel.com/download/14687/Intel-Network-Adapter-Driver-for-PCIe-Intel-10-Gigabit-Ethernet-Network-Connections-under-Linux-

Looking forward to your reply.

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

jfr · ‎06-08-2021

The screenshot was taken from the adapter user guide https://downloadmirror.intel.com/28410/eng/Adapter_User_Guide_26.3.pdf, page 35.

I did not try the latest drivers yet, I use the default driver provided in Ubuntu 20.04 (5.1.0-k).

I will try with the latest 5.11.3 and let you know if it fixed my issue.

Regards,

Julien

jfr · ‎06-08-2021

Hello Crisselle,

I have updated the ixgbe drivers to the latest version:

$ ethtool -i ens1f0 
driver: ixgbe
version: 5.11.3
firmware-version: 0x8000156d, 1.2926.0
expansion-rom-version: 
bus-info: 0000:17:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

and unfortunately the switch still sees the interface up when I do "ip link set ens1f0 down", so I suppose the link is still physically up on the NIC when it is set down with ip link.

Regards,

Julien

Caguicla_Intel · ‎06-08-2021

Hello Julien,

Thank you for the prompt reply and trying out the latest driver.

We are sorry to hear that it didn't help with the problem. Please allow us to further investigate your request. We will get back to you as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

jfr · ‎06-10-2021

Hello Crisselle,

I did some tests on all our servers (HP CL2600), they all have the same hardware, but some were purchased at a different date and came with a different firmware version. This link status detection problem does not occur on the servers that were the most recently purchased.

The problem does not occur with the following firmware:
firmware-version: 0x80000492

The problem occurs with the following firmwares:
firmware-version: 0x80000482
firmware-version: 0x8000156d, 1.2926.0 (latest intel)

I have found the exact model of the NIC, it is an "HPE CL Ethernet 10GBASE-T 2-port Intel X550 PCIe 3.0 Card", part number P01665-B21 (it is listed here: https://www.intel.com/content/dam/www/public/us/en/documents/guides/hpe-product-guide.pdf)

Best regards,

Julien

Caguicla_Intel · ‎06-13-2021

Hello Julien,

Thank you for the update.

Rest assured that we will communicate this to our engineers. We will get back to you as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

Caguicla_Intel · ‎06-13-2021

Hello Julien,

While we are communicating the update to our engineers, we hope you don't mind providing the additional information below that would help with the investigation.

1. How many adapters are affected on this issue?

2. Were the adapters purchased together with the system?

3. Will you be able to provide the PBA of the rest of the adapters that are having the issue?

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support

jfr · ‎06-13-2021

Hello Crisselle,

1. On a total of 32 servers, 19 are affected on this issue. All the NICs affected in these servers have firmware 0x80000482 (except 2 which have the latest Intel firmware 0x8000156d that I used for tests)

2. Yes, the adapters were purchased together with the servers.

3. I'm not sure how I can get the PBA remotely as these servers are in a remote datacenter. As I wrote previously, there are all the same type: "HPE CL Ethernet 10GBASE-T 2-port Intel X550 PCIe 3.0 Card", part number P01665-B21.

Best regards,

Julien

Caguicla_Intel · ‎06-14-2021

Hello Julien,

Thank you for the swift response.

We will continue investigating this request together with our higher level support. We will get back to you as soon as possible but no later than 2-3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C.

Intel® Customer Support

Caguicla_Intel · ‎06-16-2021

Hello Julien,

Thank you for the patience on this matter.

Please see below information for the update on this request after further investigation.

Based on the provided information, we have confirmed that the adapters are OEM (HPE CL Ethernet 10GBASE-T 2-port Intel X550 PCIe 3.0 Card", part number P01665-B21) and HPE should still be the best source of support. For Original Equipment Manufacturer(OEM) version of Ethernet adapters, it would be best if you can get in touch with HP for further support since they have customized drivers, firmware and software to enable or alter features on your card. They are in best position where you can seek assistance so you don’t lose features or customizations of the Ethernet adapter.

You also mentioned that the issue was observed on certain firmware and this firmware is might have been customized by 3rd party manufacturer based on their desired configuration.

Once you have confirmed with HP that this known issue, they would probably release a "Customer Advisory" similar to the link below that contains resolution (please be reminded that this is for reference only and not related to the issue reported)

https://support.hpe.com/hpesc/public/docDisplay?docId=a00099187en_us&docLocale=en_US

We highly suggest contacting HPE support for them to provide solutions regarding the issue and for further assistance.

https://support.hpe.com/hpesc/public/home

Please also accept our sincerest apologies for the delay and inconvenience that this might have caused on your end.

Feel free to let us know if you have additional questions and clarifications.

Looking forward to your reply.

We will follow up after 3 business days in case we don't hear from you.

Best regards,

Crisselle C.

Intel® Customer Support

Caguicla_Intel · ‎06-21-2021

Hello Julien,

Good day!

This is just a follow up to check if you are able to get in touch with your manufacturer for further assistance on your concern. If you have additional questions and clarifications, feel free to let us know and we will be glad to help you.

Awaiting to hear from you soon.

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C.

Intel® Customer Support

Caguicla_Intel · ‎06-24-2021

Hello Julien,

How are you doing?

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.

May you have an amazing weekend ahead and stay safe!

Best regards,

Crisselle C.

Intel® Customer Support