x722-da4 unable to set speed via 'ethtool'

[olebenny]

Hello community

I am using 2 x x722-da4 in my 'servers'.

1.

If I connect 2 of these servers back to back with fiber-cables, I get links on all channels at 1 Gb.

If I try to:

ethtool -s ens1f2 speed 10000 duplex full autoneg off

responds:

netlink error: link settings update failed
netlink error: Operation not supported

2.

If I unplug one of the SPFs and put it back in, the speed is now changed to 10Gb

3.

[root@localhost ~]# ethtool -a ens1f2
Pause parameters for ens1f2:
Autonegotiate: off
RX: off
TX: off

4.

[root@localhost ~]# ethtool -i ens1f2
driver: i40e
version: 2.17.15
firmware-version: 5.40 0x80002e40 1.2992.0
expansion-rom-version:
bus-info: 0000:b6:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[root@localhost ~]#

[Achilles_Intel]

Hello olebenny,

Thank you for posting in Intel Ethernet Communities.

We are sorry to hear about this problem you encountered. For us to further check the issue, we hope you don't mind if we ask for additional details.

1. What OS are you using?
2. How many adapters are affected by this problem?
3. Please provide the front and rear photos of the adapter for validation.

Let us know if you have further questions or need clarifications. We will reach out after 3 business days if we do not receive an update from you.

Thank you and have a nice day.

Best regards,

Achilles M.

Intel Customer Support

[olebenny]

Hello Achilles M.

1. Weare using RHEL 8.5 with below version of driver and firmware:

[root@localhost ~]# ethtool -i ens1f2
driver: i40e
version: 2.17.15
firmware-version: 5.40 0x80002e40 1.2992.0
expansion-rom-version:
bus-info: 0000:b6:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
[root@localhost ~]#

2. We have 2 x x722-da4 in each system and and are going to deliver 25+ systems to end customer.

3. Please see attached photos of the board and sfp's, that we are using.

Kind regards

Ole

[Achilles_Intel]

Hello olebenny,

Thank you for providing these information. Please allow us some time to further check on this. We will provide you with an update no later than 3 business days.

Thank you and have a nice day.

Best regards,

Achilles M.

Intel Customer Support

[Achilles_Intel]

Hello olebenny,

Greetings.

We are still currently checking on your inquiry. We hope you don't mind if you could confirm the following details for us.

1. We noticed that the adapter card does not have a heatsink installed. Was it intentionally removed or they purchased it AS IS?

2. Please provide us the Intel SSU. For reference, you can check out this link below.

https://www.intel.com/content/www/us/en/support/articles/000008563/ethernet-products.html

3. Kindly provide the ethtool -S so we could get more details from the adapter.

4. Please confirm if we understood the issue correctly:

You were unable to set the adapter to 10Gbps as you encountered an error however, once you unplugged one of the SPFs and put it back in, the speed changed to 10Gbps. Does it mean that you were able to change the speed using ethtool without an error?

Once it was changed to 10Gbps after unplugging one of the SFPs, does it automatically change back to 1Gbps so you are trying to change it back to 10Gb?

Let us know if you need clarifications. We will reach out after 3 business days should we not receive a response from you.

Thank you and have a nice day.

Best regards,

Achilles M.

Intel Customer Support

[Achilles_Intel]

Hello olebenny,

Good day.

We are following up on the details that we requested from you. Feel free to let us know if you have further questions or need clarifications. We will reach out again after 3 business days if we do not receive a response from you.

Thank you and have a nice day.

Best regards,

Achilles M.

Intel Customer Support

[olebenny]

Hi Achilles M.

So sorry, I replied directly to email, instead of here, so you never received my response.

Here it is again:

I copied your 4. Questions here and highlighted my answers in lightblue.

1. On the modules installed in our system, we do have a heatsink ( see attached picture )
2. Also see attached SSU_ISU_2xX722-DA4_24Mar2022.txt with output from ssu.sh
3. And ethtool_-S_ens1f3.txt for output of ‘ethtool -S ens1f3’

4.

You were unable to set the adapter to 10Gbps as you encountered an error however, once you unplugged one of the SPFs and put it back in, the speed changed to 10Gbps.

Ole: Correct

Does it mean that you were able to change the speed using ethtool without an error?

Ole: No, I cant change speed to something else than ‘current’

[root@localhost ~]# ethtool -s ens1f3 speed 10000                      ß current == 10000 Gb        à         No error

[root@localhost ~]# ethtool -s ens1f3 speed 1000                       ß current == 10000 Gb à   Error

netlink error: link settings update failed

netlink error: Operation not supported

[root@localhost ~]#

Once it was changed to 10Gbps after unplugging one of the SFPs, does it automatically change back to 1Gbps so you are trying to change it back to 10Gb?

Ole: No, once change to 10 Gb, it stays at 10 G

I noticed a ‘new  behaviour’:

            If I boot without fiber-cable connected, if starts at 10 Gb, but still fails to set ‘speed 1000’

Let us know if you need clarifications. We will reach out after 3 business days should we not receive a response from you.

Ole: OK

Best regards

Ole

[Achilles_Intel]

Hello olebenny,

Thank you for providing these information. Please allow us to share these with our team and further check on your concern. We will provide you with an update as soon as it is available to us but no later than 3 business days.

Thank you and have a nice day.

Best regards,

Achilles M.

Intel Customer Support

[Achilles_Intel]

Hello olebenny,

Good day.

You can try the latest driver available on sourceforge and check if you will still encounter the error.

https://sourceforge.net/projects/e1000/files/i40e%20stable/2.18.9/

If the problem still persist, kindly clarify if the step by step of the issue is correct. If there are any missing details, please let us know.

1. All 4 SFP plugged into the X722-DA4 card then run the command however it encountered an error
2. Unplugged one of the SFPs then installed back again , run the command to set the 10Gbps again, no error message and the connection stayed at 10Gbps
3. Unplugged one of the SFPs , connection stays at 10Gbps.

Also, the log files was not able to capture some important details from the adapter cards such as the PBA number. Is it possible for you to remove one of the adapter cards to provide the front and rear photo?

Let us know if you questions or need clarifications. We will make a follow-up after 3 business days if we do not receive an update from you.

Thank you and stay safe.

Best regards,

Achilles M.

Intel Customer Support

[olebenny]

Hi Achilles

I will try 'https://sourceforge.net/projects/e1000/files/i40e%20stable/2.18.9/' shortly.

The description of the problem is:

1. 2 systems, each one having one x722-da4 with 4 fiber cables connected one to one between the 2 boards.

2. power up the 2 systems with RHEL 8.5

3. Now all 4 ports in each card shows left led == yellow, right led == green ===> 1Gb speed ( pictires attached for the 2 modules )

and 'ethtool ens1f0' shows :

[root@localhost ~]# ethtool ens1f0
Settings for ens1f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
1000baseX/Full
10000baseSR/Full
10000baseLR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
1000baseX/Full
10000baseSR/Full
10000baseLR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x0000000f (15)
drv probe link timer
Link detected: yes
[root@localhost ~]#

5.  Trying to set speed to 10Gb by using 'ethtool -s ens1f0 speed 10000' gives this error:

[root@localhost ~]# ethtool -s ens1f0 autoneg off speed 100000
netlink error: link settings update failed
netlink error: Operation not supported
[root@localhost ~]#

6. Unplugging any of the SPF's, will make that channel change speed to 10 Gb. ( picture attached  )

7. But so far, I haven't been able to set 'speed' via 'ethtool -s ens1f0 speed 10000'

best regards

Best regards

[Achilles_Intel]

Hello olebenny,

Thank you for the response and providing us these details. We will wait for your update regarding the driver. We will reach out after 3 business days for a follow-up should we not receive a response from you.

Thank you and have a nice day.

Best regards,

Achilles M.

Intel Customer Support

[Achilles_Intel]

Hello olebenny,

Good day.

We are just checking if the suggestion we provided was able to help you with your inquiry. If you have questions or concerns, feel free to inform us. We will reach out again after 3 business days should we not receive an update from you.

Thank you and have a nice day.

Best regards,

Achilles M.

Intel Customer Support

[Mike_Intel]

Hello olebenny,

We am just sending another follow up on the information that I requested. 

Since we have not heard back from you, I need to close this inquiry. 

If you need any additional information, please submit a new question as this thread will no longer be monitored.

You may receive an invitation to take a survey in a few days. We value your feedback and look forward to hearing about your support experience.

Best regards,

Michael L.

Intel® Customer Support Technician

[olebenny]

Hello Michael

1.

The new driver does not make any difference.

2.

When the systems is powered up, they seem to start as 1 Gb, and the still work with ping and so on.

Power up settings ( 1 led yellow ( speed ) and the other green( activity )

[root@localhost ~]# ethtool ens1f0
Settings for ens1f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
1000baseX/Full
10000baseSR/Full
10000baseLR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
1000baseX/Full
10000baseSR/Full
10000baseLR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: off
Port: FIBRE
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x0000000f (15)
drv probe link timer
Link detected: yes
[root@localhost ~]#

3.

As I need to use them at 10 Gb, the only way right now to change speed, is to pull out SFP and re-seat it again.

This makes both leds green per channel.

4.

Help

Best regards

Ole

[Mike_Intel]

Hello olebenny,

Thank you for the update. We will try to check this issue again and get back to on how we can proceed with the issue. Please give us 2 to 3 working days to provide an update on this thread.

Best regards,

Michael L.

Intel® Customer Support Technician

[Mike_Intel]

Hello olebenny,

Thank you for patiently waiting for our update. And we just want to inform you that it has been verified that the X722 cards do not work with multi-rate optic modules. The only supported configuration is single rate modules. This can also be found on product brief. Intel® Ethernet Network Adapter X722 page 3

https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ethernet-network-adapter-x722-product-brief.pdf

Also, the multi-rate modules speed cannot be changed using ethtool in that way. We highly recommend that you use Single rate 10Gb modules with these cards.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support 

[olebenny]

Hi Michael

I did read the product-brief for x722-da4, but the people before me on this project did not......

And I have to admit, that the x722-da4 do work with dual-speed spf+ modules:

1.

If I connect 2 x722-da4 'back to back' in 2 different systems, all 4 channels start up with 1Gbit links ( speed is yellow ).

Assigning different ip-addresses ( on different subnets) to each channel on x722-da4 in the 2 systems, I can ping the other end at 1Gbit.

If I unplug one of the spf+-modules in one of the x722-da4 - modules and re-seat it again, it now connects at 10Gbit, and I can now ping at 10Gbit on that channel, while still pinging at 1 Gbit on the other channels.

So the card works on 1Gbit and 10Gbit hardware wise, but the software does not support it for setting the speed, because product-brief for x722-da4 says it does not support it.

2.

Anyway, we have another problem, and that is availability and 'EOL' for the x722-da4.

Which Intel module can replace the X722-DA4 and is available ????

3.

And thank you for the excellent support so far.

Best regards

Ole

[olebenny]

Hi Michael

I did in fact take a picture of the screen, where I run 1 channel on the x722-da4 on 1Gbit and another one at 10Gbit with 'iperf3', showing that they perform as expected ( see attached photo ).

Best regards

Ole

[olebenny]

Hi Michael

I did find this Intel module ( x722-da4):

https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ethernet-x710-brief.pdf

It seems to have similar specs or better.

Do this module also have problems with dual-speed sfp+-modules ??

Best regards

Ole

[Mike_Intel]

Hello olebenny,

I need to further check all of these questions that you asked. Please give us 2 to 3 working days to post an update. Thank you.

Best regards,

Michael L.

Intel® Customer Support 

[olebenny]

Hi Michael

Sounds great, thanks.

Best regards

Ole