Question about Simultaneous Port Usage on the XL710

WaugiCloud · ‎07-03-2024

Hello,

Question: On the XL710 (40G) card, if I connect 2 40Gx2 optical fibers, can I use 40G on each port at the same time, or can I only use 20 Gbps on each? How did this work for you?

01:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

ethtool -i ens1f1
driver: i40e
version: 2.25.7
firmware-version: 7.00 0x80004ca8 255.65535.255
expansion-rom-version:
bus-info: 0000:01:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Thank you

ManoranjanDas · ‎07-05-2024

Hi Alan,

Greetings for the day!

Based on your query, please provide the following details to proceed further:

1. Full system/server model details.

2. Details of the Ethernet card. (Including clear pictures from both sides if possible)

3. Kindly confirm whether the Intel Ethernet controller XL710 was included with the system or added later.

4. Are there any additional issues you are facing?

Thank you for your cooperation and support.

Best Regards,

Manoranjan.

WaugiCloud · ‎07-05-2024

Hola @ManoranjanDas

Thank you for your response. Here is some context. I have a Supermicro server with an X10DRi motherboard, two Intel Xeon E5-2683 v4 @ 2.10GHz processors, and the Intel XL710 (40Gx2) network card.

I don't have photos of the XL710 right now, as it's in production and connected to the server. To take photos of both sides, I would need to disconnect it or figure out how to do so without interrupting the service.

I manually installed the i40e drivers for the XL710 (make install) by downloading them from Intel's official website. The version I currently have is 2.25.7, and the NVM (firmware version) is also updated: 7.00 0x80004ca8 255.65535.255.

Now that I've given you some context, here is my problem: I connected two 40Gx2 optical fiber ports to the two available ports on the XL710, but I noticed that I cannot exceed 19 Gbps on port 1, while on port 0, I have not seen a precise limit (as I have not moved much traffic through that port yet), although I reached 10 Gbps.

In summary, on port 0, I reached about 10 Gbps. On port 1, I reached about 19 Gbps and began experiencing packet loss, without observing any IRQ at 100%. That is, everything seemed normal in terms of processor usage.

Port 0 (ens1f0)

Port 1 (ens1f1)

My question is as follows: Is it possible to achieve 80 Gbps, that is, 40 Gbps on each of the two ports, but not in bonding mode, instead with different traffic on each port? Is there a PCI bus limitation? I ask because when adding the traffic from ports 0 and 1, I only managed to reach about 30 Gbps approximately.

I have HT and virtualization disabled as recommended by Intel in the guide for 700 series network cards for optimization in Linux.

# ethtool -i ens1f0
driver: i40e
version: 2.25.7
firmware-version: 7.00 0x80004ca8 255.65535.255
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

ethtool -g ens1f0
Ring parameters for ens1f0:
Pre-set maximums:
RX:             4096
RX Mini:        n/a
RX Jumbo:       n/a
TX:             4096
Current hardware settings:
RX:             256
RX Mini:        n/a
RX Jumbo:       n/a
TX:             256
RX Buf Len:             n/a
CQE Size:               n/a
TX Push:        off
TCP data split: n/a

# ethtool -l ens1f0
Channel parameters for ens1f0:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       32
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       32

ethtool -x ens1f0
RX flow hash indirection table for ens1f0 with 32 RX ring(s):
    0:      0     0     0     0     0     0     0     0
    8:      0     0     0     0     0     0     0     0
   16:      1     1     1     1     1     1     1     1
   24:      1     1     1     1     1     1     1     1
   32:      2     2     2     2     2     2     2     2
   40:      2     2     2     2     2     2     2     2
   48:      3     3     3     3     3     3     3     3
   56:      3     3     3     3     3     3     3     3
   64:      4     4     4     4     4     4     4     4
   72:      4     4     4     4     4     4     4     4
   80:      5     5     5     5     5     5     5     5
   88:      5     5     5     5     5     5     5     5
   96:      6     6     6     6     6     6     6     6
  104:      6     6     6     6     6     6     6     6
  112:      7     7     7     7     7     7     7     7
  120:      7     7     7     7     7     7     7     7
  128:      8     8     8     8     8     8     8     8
  136:      8     8     8     8     8     8     8     8
  144:      9     9     9     9     9     9     9     9
  152:      9     9     9     9     9     9     9     9
  160:     10    10    10    10    10    10    10    10
  168:     10    10    10    10    10    10    10    10
  176:     11    11    11    11    11    11    11    11
  184:     11    11    11    11    11    11    11    11
  192:     12    12    12    12    12    12    12    12
  200:     12    12    12    12    12    12    12    12
  208:     13    13    13    13    13    13    13    13
  216:     13    13    13    13    13    13    13    13
  224:     14    14    14    14    14    14    14    14
  232:     14    14    14    14    14    14    14    14
  240:     15    15    15    15    15    15    15    15
  248:     15    15    15    15    15    15    15    15
  256:     16    16    16    16    16    16    16    16
  264:     16    16    16    16    16    16    16    16
  272:     17    17    17    17    17    17    17    17
  280:     17    17    17    17    17    17    17    17
  288:     18    18    18    18    18    18    18    18
  296:     18    18    18    18    18    18    18    18
  304:     19    19    19    19    19    19    19    19
  312:     19    19    19    19    19    19    19    19
  320:     20    20    20    20    20    20    20    20
  328:     20    20    20    20    20    20    20    20
  336:     21    21    21    21    21    21    21    21
  344:     21    21    21    21    21    21    21    21
  352:     22    22    22    22    22    22    22    22
  360:     22    22    22    22    22    22    22    22
  368:     23    23    23    23    23    23    23    23
  376:     23    23    23    23    23    23    23    23
  384:     24    24    24    24    24    24    24    24
  392:     24    24    24    24    24    24    24    24
  400:     25    25    25    25    25    25    25    25
  408:     25    25    25    25    25    25    25    25
  416:     26    26    26    26    26    26    26    26
  424:     26    26    26    26    26    26    26    26
  432:     27    27    27    27    27    27    27    27
  440:     27    27    27    27    27    27    27    27
  448:     28    28    28    28    28    28    28    28
  456:     28    28    28    28    28    28    28    28
  464:     29    29    29    29    29    29    29    29
  472:     29    29    29    29    29    29    29    29
  480:     30    30    30    30    30    30    30    30
  488:     30    30    30    30    30    30    30    30
  496:     31    31    31    31    31    31    31    31
  504:     31    31    31    31    31    31    31    31
RSS hash key:
ff:87:d9:a1:43:3c:36:e1:5c:98:92:5a:87:e8:de:6f:91:fe:80:eb:30:3e:5f:3d:58:3c:27:78:6d:59:ba:f7:5f:d3:28:aa:a9:0c:a1:33:dd:17:f2:fc:99:fe:5f:b0:7c:03:6e:d6
RSS hash function:
    toeplitz: on
    xor: off
    crc32: off

Best Regards,

Alan

Azeem_Intel · ‎07-05-2024

Hi WaugiCloud,

Thank you for posting in Intel Ethernet Communities.

Please provide snapshots of both sides of the Ethernet card with the serial number visible for further validation. Also, since we have limited details about your Ethernet product to offer the best solution, have you attempted to install the latest driver version available from the Intel download site?

Intel® Network Adapter Driver for PCIe* 40 Gigabit Ethernet Network Connections under Linux*

https://www.intel.com/content/www/us/en/download/18026/intel-network-adapter-driver-for-pcie-40-gigabit-ethernet-network-connections-under-linux.html

Regards,

Azeem

WaugiCloud · ‎07-05-2024

Hello @Azeem_Intel

You didn't read what I wrote, did you? I mentioned that the XL710 card is in production. To be able to take pictures of the card, I would need to disconnect it from the PCI, and for that, I would need to schedule a maintenance instance. I can't turn off, disconnect, and take pictures at this moment.

I will see about doing it, but for now, it's unfeasible without scheduling maintenance.

On the other hand, I haven't tried the latest drivers yet. The ones I have now are the previous ones, and the latest ones were released last month. I doubt they will solve the problem, but I will try them.

Answering my query, is it possible or not to use 80 Gbps? 40 Gbps on each port? Or is there some limitation?

Regards and thanks.

Azeem_Intel · ‎07-05-2024

Hi WaugiCloud,

Greeting for the day!

Sorry for the inconvenience. There are different models within the XL710 series, and we want to ensure we provide accurate support for your specific product. However, I will share some general details that may be helpful:

For example, in one model, the Data Rate Per Port is 40/10GbE. If you are sure this is the same model, then the information will apply. Additionally, please let us know if you still encounter any issues after updating to the latest driver.

https://ark.intel.com/content/www/us/en/ark/products/83967/intel-ethernet-converged-network-adapter-xl710-qda2.html

Feel free to reply to this email, and we'll be more than happy to assist you further.

Regards,

Azeem

ManoranjanDas · ‎07-08-2024

Hello WaugiCloud,

Greeting for the Day!

We are currently awaiting your response regarding the case. If you have any queries or require further assistance, please feel free to respond on the community post. We are more than happy to assist you.

Thank you for choosing Intel products and services.

Regards,

Manoranjan Das.

ManoranjanDas · ‎07-11-2024

Hello Alan,

Greeting for the Day!

We are currently awaiting your response regarding the case. If you have any queries or require further assistance, please feel free to respond on the community post. We are more than happy to assist you.

Thank you for choosing Intel products and services.

Regards,

Manoranjan.

WaugiCloud · ‎07-11-2024

Hello @ManoranjanDas @Azeem_Intel

I still haven't been able to schedule a maintenance window. I would like to schedule one to remove the server board, take photos of both sides, and send them to you. I installed (make install) the latest i40e-2.25.9 drivers on the Intel XL710 board, but I still haven't been able to put them into production. I will try to perform the maintenance task and put them into production next week.

I noticed that the drivers are not compatible with Debian 11 or Debian 12. The same thing happened with an Intel E810-C, and I had to revert to Ubuntu 22.04 (Jammy) where the drivers compile correctly without symbolic link warnings. The same thing happened with the XL710 on Debian 12 and Debian 11, and also with the Intel E810-C. Therefore, I will schedule a maintenance task to take photos of the XL710 board and switch to Ubuntu 22.04, where the drivers compiled with practically no warnings.

Using Debian 12

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm

With the production drivers of the Intel XL710: (2.25.7) and the updated firmware: 7.00

# ethtool -i ens1f0
driver: i40e
version: 2.25.7
firmware-version: 7.00
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

I noticed a loss of performance (lost packets) when using both 40 Gbps x2 ports simultaneously, especially on the ens1f1 port, where I didn't exceed 17 Gbps, while on the ens1f0 port I reached 10 Gbps. Simultaneously, both ports reached 27 Gbps (10+17 Gbps), as I mentioned, without IRQ interruption issues. Could it have something to do with the baseSR4/Full or baseCR4/Full cable?

 ethtool ens1f0
Settings for ens1f0:
        Supported ports: [ FIBRE ]
        Supported link modes:   40000baseSR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  40000baseSR4/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x0000000f (15)
                               drv probe link timer
        Link detected: yes

 ethtool ens1f1
Settings for ens1f1:
        Supported ports: [ FIBRE ]
        Supported link modes:   40000baseCR4/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  40000baseCR4/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 40000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: Direct Attach Copper
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: g
        Wake-on: g
        Current message level: 0x0000000f (15)
                               drv probe link timer
        Link detected: yes

I am going to schedule the maintenance window to switch to Ubuntu 22.04 and redirect traffic to the ens1f1 port to see how both operational ports react, and if I continue to notice the degradation of lost packets without any core at 100%/IRQ.

Is there a way to know the exact model of the XL710 from the console, or do I have to go to the data center to disconnect it?

I'll keep you updated, and if you know anything, I'm looking forward to your comments.

RyanFeeko · ‎07-11-2024

hey @WaugiCloud regarding your inquiry: "is there a way to know the exact model of the XL710 from the console, or do I have to go to the data center to disconnect it?"

kindly try this command below:

sudo lshw -class network -short
or
lshw -class network

ManoranjanDas · ‎07-15-2024

Hi Alan,

Greetings for the day!

Thank you for sharing your findings regarding the Intel XL710 network cards and the performance challenges on Debian 11 and 12. Moving forward, transitioning to Ubuntu 22.04 to resolve driver compatibility issues is a proactive step. We appreciate your scheduled maintenance to evaluate ens1f1 port performance.

Please keep us updated on your progress. If you have any further questions or need assistance, don't hesitate to reach out.

Thank you for choosing Intel products and services.

Best regards,

Manoranjan.

ManoranjanDas · ‎07-19-2024

Hello Alan,

Greetings for the day!

I hope this message finds you well.

We are following up on this case. Please reply to confirm the start of your system status so we can continue to help with a resolution. We look forward to receiving your reply.

Thank you for choosing Intel products and services.

Regards,

Manoranjan.

ManoranjanDas · ‎07-22-2024

Hello Alan,

Greeting for the Day!

We are currently awaiting your response regarding the case. If you have any queries or require further assistance, please feel free to respond on the community post. We are more than happy to assist you.

Thank you for choosing Intel products and services.

Regards,

Manoranjan.

ManoranjanDas · ‎07-26-2024

Hello WaugiCloud,

Greetings for the day!

We would like to inform you that we are closing this request as no response has been received from our previous follow-ups.

Please don't hesitate to ask any further questions in the future. Feel free to start a new conversation, as this thread will no longer be monitored.

Regards,

Manoranjan.