XL710 configure physical port ETS

[Matevz]

Hi,

we are trying to program XL710 physical port to allow 8 strict priority TCs, Linux opensource driver or Intel driver, does not matter, same behviour. NVM 6.01

The following code:

        memset(&ets_data, 0, sizeof(ets_data));

        ets_data.tc_valid_bits = 0xff;

        ets_data.tc_strict_priority_flags = 0xff;

        ets_data.tc_bw_share_credits[0] = 127;

        ets_data.tc_bw_share_credits[1] = 127;

        ets_data.tc_bw_share_credits[2] = 127;

        ets_data.tc_bw_share_credits[3] = 127;

        ets_data.tc_bw_share_credits[4] = 127;

        ets_data.tc_bw_share_credits[5] = 127;

        ets_data.tc_bw_share_credits[6] = 127;

        ets_data.tc_bw_share_credits[7] = 127;

        ret = i40e_aq_config_switch_comp_ets(hw,

              pf->mac_seid,

              &ets_data,

              i40e_aqc_opc_enable_switching_comp_ets,

              NULL);

works and returns no error, but TC Strict priority filed stays at 0 and TCs are not getting scheduled in SP fashion, but Round robin.

Does XL710 need to operate in any special mode to get SP scheduling working? Or any other idea what might be wrong with it.

Matevz

[AlfredoS_Intel]

Hi Matevz,

Thank you for posting on our Intel® Ethernet Communities Page.

We are sorry to hear about the issue that you are experiencing with your network adapter. 

So we would have a better understanding of your issue, please provide the following information:

1. Please download and run our Intel® System Support Utility from this page, https://downloadcenter.intel.com/download/26735/Intel-System-Support-Utility-for-the-Linux-Operating-System#:~:text=Intel%20SSU%20for%20the%20Linux,and%20shared%20by%20the%20user. After running it, you will be given an option to save the logs to a text file, please do so and attach the file on your reply.

2. Kindly provide us the results of this command: ethtool -i ethx where ethx is the Ethernet port.

3. Please provide the exact model of the ethernet XL710 adapter

We look forward to hearing from you. If we do not get your reply, we will follow up after 3 business days.

Best Regards,

Alfred S

Intel® Customer Support

[Matevz]

Hi,

thanks for your answer. Actually this is embedded hardware, but the issue is the same on adapter:

driver: i40e

version: 2.3.6

firmware-version: 6.01 0x800034aa 0.0.0

expansion-rom-version: 

bus-info: 0000:01:00.2

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

Embedded variant:

Intel Corporation Ethernet Controller X710 for 10GbE backplane (rev 02)

Adpater: 

Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

We have managed to resolve this particular issue, firmware is no longer replying with error code. But the general problem remains.

We are trying to create a very simple configuration where XL710 would be used in Linux in non DCB mode (DCB not build into Linux) and the following configuration:

PF->VEB->main VSI (which is default i40e config anyway)

+ 8 egress TCs organized in strict priority.

We were able to create 8 TCs, 8 Qsets, 8 Qpairs, but the problem is in egress we are getting round-robin servicing instead of required strict priority.

We were posting required TX Scheduler AQ commands setting Port in ETS, enabling TCs, ... but nothing helps.

We would like to keep the default configuration created by i40e driver as much as possible and only enable 8 TCs with SP servicing on default VSI.

We are able to modify kernel driver source code.

Do you have any step by step guide what all must be performed to correctly enable SP TX scheduling?

We set:

AQ  0x0413 (for PF), TC0-TC7 Valid=0xFF, TC0-TC7 Strict Priority Flag=0x7f,  TCx Bandwidth Credits = 0x7f for all TCs

AQ 0x0407 (for main VSI),  TC0-TC7 Valid=0xFF, ,  TCx Bandwidth Credits = 0x7f for all TCs

XL710 FW says all OK, but in reality it does not perform SP scheduling.

It behaves like ETS scheduler settings are not used at all.

We have tried disabling LLDP in XL710, it did not get us to SP, but at least there was a change seen after that.

Thanks,

Matevz

[Matevz]

Dear Intel,

bellow please find dump of 5 Query Scheduler AQ commands displaying our configuration of the scheduler. Does this seem OK to you for the required setup?

7.8.4.15 Query VSI Bandwidth Configuration
KOZA: KOZA DUMPING response for i40e_aq_query_vsi_bw_config SEID 398 err 0
raw data: ffff000014cfbcb0: ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbcc0: 02 00 14 00 15 00 16 00 17 00 18 00 19 00 1a 00 ................
raw data: ffff000014cfbcd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

7.8.4.16 Query VSI Bandwidth Configuration per Traffic Type
KOZA: KOZA DUMPING response for i40e_aq_query_vsi_ets_sla_config SEID 398 err 0
raw data: ffff000014cfbc30: ff 00 00 00 7f 7f 7f 7f 7f 7f 7f 7f 00 00 00 00 ................
raw data: ffff000014cfbc40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

7.8.4.16 Query VSI Bandwidth Configuration per Traffic Type
KOZA: KOZA DUMPING response for i40e_aq_query_switch_comp_ets_config SEID 162 err 0
raw data: ffff000014cfbc70: ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbc80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbc90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

7.8.4.18 Query Physical Port ETS Configuration
KOZA: KOZA DUMPING response for i40e_aq_query_port_ets_config SEID 4 err 0
raw data: ffff000014cfbcf0: 00 00 00 00 ff 00 7f 00 7f 7f 7f 7f 7f 7f 7f 7f ................
raw data: ffff000014cfbd00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbd10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbd20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
raw data: ffff000014cfbd30: 00 00 00 00 ....

7.8.4.19 Query Switching Component Bandwidth Configuration per Traffic Type
KOZA: KOZA DUMPING response for i40e_aq_query_switch_comp_bw_config SEID 162 err 0
raw data: ffff000014cfbc50: ff 00 00 00 7f 7f 7f 7f 7f 7f 7f 7f 00 00 00 00 ................
raw data: ffff000014cfbc60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

DUMP SWITCH topology:
i40e 0000:01:00.2: header: 3 reported 3 total
i40e 0000:01:00.2: type=19 seid=402 uplink=162 downlink=18
i40e 0000:01:00.2: type=17 seid=162 uplink=4 downlink=398
i40e 0000:01:00.2: type=19 seid=398 uplink=162 downlink=18  -> MAIN VSI

regards,

Matevz

[AlfredoS_Intel]

Hi Matevz,

Thank you providing those information.

Please allow us some time to check on this. 

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support 

[Matevz]

Hi Alfred,

we have replaced XL710 firmware with the latest one 8.15 (previously we were using 6.10) and things started working much much better now. That was actually the only change.

We are now having a setup where we send out traffic at 9 Gbit/s, 4.5Gbit/s to one TC and 4.5Gbit/s to another TC. Link partner is blocking XL710 transmit to 8.6 Gbit/s with PAUSE frames.

In this case XL710 is correctly prioritising the traffic. So we get 4.5Gbit/s of higher priority traffic and 4,1Gbit/s of lower priority.

If we have higher priority traffic 1Gbit/s and lower priority traffic 8 Gbit/s, then we get all higher and 7.6Gbit/s lower. Which is also FINE.

But if we have 8 Gbit/s higher priority and 1 Gbit/s lower priority, only 8Gbit/s stream will suffer, but it should not as it is higher priority stream.

So it seems it is still not strict priority scheduling as it should be.

thanks,

Matevz

[AlfredoS_Intel]

Hi Matevz,

Thank you for your response and providing more details regarding the issue,

We will continue checking this and the information that provided is invaluable to us.

Please allow us some time to check on this. We hope for your understanding regarding this

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support 

[AlfredoS_Intel]

Hi Matevz,

We are just following up.

It looks like you need more time to provide the information that we asked.

We will follow up again after 3 business days. Should we not hear from you, our system may automatically close the thread.

Best Regards,

Alfred S

Intel Customer Support

[Matevz]

Hi Alfred,

actually we are waiting your feedback. There are no questions we have not answered here.

Maybe you entered some questions but marked the reply as private/internal or something like that.

best regards,

Matevz

[AlfredoS_Intel]

Hi Matevz,

Thank you for your response. Please accept our humblest apologies for the confusion it seems one our posts did not reflect on the thread.

Our team would like to get more information on your system:

Please download and run our Intel® System Support Utility from this page, https://downloadcenter.intel.com/download/26735/Intel-System-Support-Utility-for-the-Linux-Operating-System#:~:text=Intel%20SSU%20for%20the%20Linux,and%20shared%20by%20the%20user. After running it, you will be given an option to save the logs to a text file, please do so and attach the file on your reply.

We also need to get the PBA from your card. We need to determine if it is a retail adapter. Are you able to provide us with pictures of the markings printed on the stickers on the card? If yes, kindly provide us pictures focusing on the characters printed on the sticker and the PBA number printed on the circuit board of the card.

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[Matevz]

Hi Alfred,

OK, but please try to prioritise it now, this already got critically urgent.

Attached ssu log.

This is not an adapter but chip directly integrated into embedded product. Flash was programmed with the following contents XL710_X710_8p15_SFI_Backplane_MCTP_CFGID7p0_Quad.bin

The file was included in NVM update 8.15 obtained from https://downloadcenter.intel.com/download/24769/Non-Volatile-Memory-NVM-Update-Utility-for-Intel-Ethernet-Network-Adapter-700-Series

best regards,

Matevz

[Mike_Intel]

Hello Matevz,

Thank you for sending the SSU, we will read and check this SSU logs that you provided.

While doing that, please provide the part number of the network card that we also requested.

If you have questions, please let us know. In case we do not hear from you, we will make a follow up after 3 workings days. Thank you.

Best regards,

Michael L.

Intel® Customer Support

[Matevz]

Hi,

it is not network card, but chip itself.

partnum:

ITLFTXL710BM1SLLK9

regards,

Matevz

[AlfredoS_Intel]

Hi Matevz,

Thank you for your response and for the time that you spent to provide the part number.

On one of your previous replies, you mentioned that you also have a network adapter version. May we know the PBA number on that card, as that is the information that will help us in further checking your issue?

For reference in looking for the part number you may check this link,  https://www.intel.com/content/www/us/en/support/articles/000005625/services.html.

Additionally, you may also check this link, https://www.intel.com/content/www/us/en/support/articles/000007060/network-and-i-o/ethernet-products.html.

We would also like to ask, on the SSU logs that you have provided, it seems you are using the built-in version of the card, to confirm, are we correct with this finding?

We look forward to your reply. Should we not get your reply, we will follow up after three business days.

Best Regards,

Alfred S

Intel® Customer Support

[Matevz]

Hi,

there is no PBA number. We use your chip soldered directly to our mainboard.

Yes, this is built-in adapter.

regards,

Matevz

[AlfredoS_Intel]

Hi Matevz,

Thank you for your response and for confirming what adapter you are using.

We will continue checking this and the information that provided is invaluable to us.

Please allow us some time to check on this. We hope for your understanding regarding this

We will get back to you no later than 3 business days from now.

Best Regards,

Alfred S

Intel® Customer Support

[CarlosAM_INTEL]

Hello, @Matevz:

Thank you for contacting Intel Embedded Community.

You should verify that the affected Intel® Ethernet Controller XL710 has the proper NVM. You can find it in the Intel® Ethernet Controller XL710/X710 production NVM images 6.80 document # 564666, which is accessible when you are logged into your Resource and Design Center (RDC) privileged account on the following website:

https://cdrdv2.intel.com/v1/dl/getContent/564666

You should fill out the form stated on the following website when you have problems with the provided website or want to update your RDC account:

https://www.intel.com/content/www/us/en/forms/support/my-intel-sign-on-support.html

Best regards,

@CarlosAM_INTEL.

[Matevz]

Hi Carlos,

we are using file XL710_X710_8p15_SFI_Backplane_MCTP_CFGID7p0_Quad.bin which we believe is the correct one. Initial production programming was done with external tool writing this image directly to Flash chip while Intel XL710 was held in reset.

Is this the correct procedure?

Is the file XL710_X710_8p15_SFI_Backplane_MCTP_CFGID7p0_Quad.bin correct for required Strict priority configuration?

Is Strict priority configuration supported in this NVM?

Is there an application note or list of correct AdminQ commands required to configure scheduler into SP mode? We have performed configuration based on XL710 Datasheet, but perhaps there is some documentation available for each firmware version as well as we have noticed different NVM images behave differently.

regards,

Matevz

[CarlosAM_INTEL]

Hello, @Matevz:

Thanks for your reply.

You should use the Intel® Ethernet XL710 10 GbE 40 GbE Controller Fortville ESX 5.5 Driver Beta Version 1.6.6 document # 567913 that may solve the reported inconvenience. You can find it when you are logged into your Resource & Design Center (RDC) privileged account on the following website:

https://cdrdv2.intel.com/v1/dl/getContent/567913

You should fill out the RDC Account Support form to process your account update request or report any inconveniences with the provided site. You can be found on the following website:

https://www.intel.com/content/www/us/en/forms/support/my-intel-sign-on-support.html

Best regards,

@CarlosAM_INTEL.

[Matevz]

Hi,

this is ESX driver in binary form. Do you have it available in source format?

Ideally for Linux?

Linux i40e which is available does not include advanced QoS configuration features.

regards,

Matevz

[CarlosAM_INTEL]

Hello, @Matevz:

Thanks for your reply.

You should address your request as a reference to the channels listed on the following website:

https://01.org/community

Best regards,

@CarlosAM_INTEL.