topic Re: Re:E810-XXV PFC in Ethernet Products

E810-XXV PFC

EvgeniyL — Mon, 18 Mar 2024 21:10:19 GMT

I have server with Intel(R) Ethernet Controller E810-XXV for SFP.

Server working on VMware 7.0.3. I want use RDMA for vSAN, but i see failed configuratiob - The PFC is not enabled.

Now
Mode: 3 - IEEE Mode
PFC Enabled: false

I want set PFC Enabled in true. How to set PFC Enabled values to true?

Re:E810-XXV PFC

vij1 — Tue, 19 Mar 2024 13:18:45 GMT

Hello Evgeniyl,

Greetings for the day!

For using RDMA, download the irdman driver from the VMware site. After loading the RDMA driver(irdman), the NIC will be automatically loaded in ROCEv2 mode.

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

Refer to "Driver setup on Host" section in irdman driver release notes for detailed explanation on how switching between ROCEv2 and iWARP happens.

Regards,

Vijay N.

Re: Re:E810-XXV PFC

EvgeniyL — Tue, 19 Mar 2024 14:40:29 GMT

Hi Vijay,

i use latest driver for icen and irdman:

esxcli software vib list |grep icen
icen 1.12.5.0-1OEM.702.0.0.17630552 INT VMwareCertified 2024-02-08

esxcli software vib list |grep ird
irdman 1.4.3.0-1OEM.700.1.0.15843807 INT VMwareCertified 2024-02-08

This is my model my card

VMware Compatibility Guide - I/O Device Search

recommended drivers

ESXi 7.0 U3 icen version 1.12.5.0-1OEM
ESXi 7.0 U3 irdman version 1.4.3.0

When i check PFC, PFC

esxcli network nic dcb status get -n vmnic5
Nic Name: vmnic5
Mode: 3 - IEEE Mode
Enabled: true
Capabilities:
Priority Group: true
Priority Flow Control: true
PG Traffic Classes: 8
PFC Traffic Classes: 8
PFC Enabled: false
PFC Configuration: 0 0 0 0 0 0 0 0
IEEE ETS Configuration:
Willing Bit In ETS Config TLV: 0
Supported Capacity: 8
Credit Based Shaper ETS Algorithm Supported: 0x0
TX Bandwidth Per TC: 0 0 0 0 0 0 0 0
RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
TSA Assignment Table Per TC: 0 0 0 0 0 0 0 0
Priority Assignment Per TC: 0 0 0 0 0 0 0 0
Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
IEEE PFC Configuration:
Number Of Traffic Classes: 8
PFC Configuration: 0 0 0 0 0 0 0 0
Macsec Bypass Capability Is Enabled: 0
Round Trip Propagation Delay Of Link: 0
Sent PFC Frames: 0 0 0 0 0 0 0 0
Received PFC Frames: 0 0 0 0 0 0 0 0
DCB Apps:
App Type: L2 Ethertype
Protocol ID: 0x8906
User Priority: 0x3

App Type: L2 Ethertype
Protocol ID: 0x0
User Priority: 0x0

Re:E810-XXV PFC

Meghak — Tue, 19 Mar 2024 16:03:30 GMT

Hello Evgeniyl,

Greetings for the day!

Thank you for confirming the details.

Please allow us some time to review the details shared and we will get back with an update as soon as possible.

Regards,

Megha K

Re:E810-XXV PFC

vij1 — Sun, 24 Mar 2024 10:36:07 GMT

Hello Evgeniyl,

Good day!

Please find below the link for enabling PFC:

Intel® Ethernet 800 Series Linux Flow Control Configuration Guide for RDMA Use Cases:

[Link to Example 1: 800 Series Back-to-Back PCP PFC with Single TC](https://edc.intel.com/content/www/us/en/design/products/ethernet/800-series-linux-flow-control-configuration-guide-for-rdma-use-c/example-1-800-series-800-series-back-to-back-pcp-pfc-with-single-tc/)

[Link to Option A: PCP PFC Set Priorities for Drop or No Drop](https://edc.intel.com/content/www/us/en/design/products/ethernet/800-series-linux-flow-control-configuration-guide-for-rdma-use-c/option-a-pcp-pfc-set-priorities-for-drop-or-no-drop/)

Best regards,

Vijay N.

Re: Re:E810-XXV PFC

EvgeniyL — Mon, 25 Mar 2024 11:43:26 GMT

Hi Vijay,

Thanks, i saw 1 link early but this both link not for VMware. My problem with VMware and this card. How enabling PFC on ESXi ?

Re:E810-XXV PFC

vij1 — Mon, 25 Mar 2024 13:34:11 GMT

Hello Evgeniyl,

Greetings for the day!

Thank you for sharing the datails. We are checking on this issue and will get back as soon as possible.

Regards,

Vijay N.

Re:E810-XXV PFC

vij1 — Thu, 28 Mar 2024 09:34:10 GMT

Hello Evgeniyl,

Greetings for the day!

Thank you for sharing the datails. We are checking on this issue and will get back as soon as possible.

Regards,

Vijay N.

Re:E810-XXV PFC

vij1 — Thu, 04 Apr 2024 07:46:42 GMT

Hello Evgeniyl,

Greetings for the day!

DCB configuration is part of the DCB supported switch to which E810 NIC is connected to and is not directly performed in ESXi host.

Note that DCB configuration in Intel NIC is done though DCBX protocol running in SW (dcbx agent that is part ESXi). Where, there will be message exchanges between the link partners to converge on the configuration. Intel NIC is in willing mode, meaning it will accept the configuration coming from the link partner. So, all the DCB configuration needs to be done in the DCB supported switch (link partner) to get it synchronized with E810 NIC.

Regards,

Vijay N.

Re: Re:E810-XXV PFC

EvgeniyL — Thu, 04 Apr 2024 09:54:36 GMT

Hi Vijay,

We use in our infrastructure switch CISCO N9K with config:

lldp dcbx version ieee
switchport mode trunk
switchport trunk allowed vlan 1- 1024
priority-flow-control mode on
priority-flow-control watch-dog-interval on
spanning-tree port type edge trunk
mtu 9216
logging event port link-status
logging event port trunk-status
service-policy type qos input ROCE_POLICY
service-policy type queuing output ROCE_QUEUING_OUT
no shutdown

i create cluster on E810 NIC and have warning

i have host on this switch with network card Mellanox Technologies MT27800 Family [ConnectX-5] and i havent problem with PFC

Problem only with Intel(R) Ethernet Controller E810-XXV for SFP !

Re:E810-XXV PFC

vij1 — Thu, 04 Apr 2024 16:08:56 GMT

Hello Evgeniyl,

Greetings for the day!

Thank you for sharing the datails. We are checking on this issue and will get back as soon as possible.

Regards,

Vijay N.

Re:E810-XXV PFC

Poojitha — Fri, 12 Apr 2024 12:04:06 GMT

Hi EvgeniyL,

Good day!

We would like to inform you that the method to manually configure the DCB PFC using driver-specific parameters. To use this method, please check the vendor-specific driver documentation.

For example, in the icen driver, you can set the PFC priority value to 3 by running the following command and rebooting the host:

#esxcli system module parameters set -m icen -p "pfctx=0x08, pfcrx=0x08"

esxcli system module parameters set: -m icen -p "pcp_force=3"

Once you run the above commands, reboot the system.

Also, we recommend you check the switch configuration:

dcb priority-flow-control enable

dcb priority-flow-control priority 3: enable

interface ethernet 1/1-1/6 dcb priority-flow-control mode on force

By default, all the configuration for DCP on E810 is acquired from the switch configuration.

Regards,

Poojitha

Re: E810-XXV PFC

vij1 — Wed, 17 Apr 2024 06:51:31 GMT

Hello EvgeniyL,

Greetings for the day!

We hope this message finds you well. We are sending a follow up if you have any questions from the previous post. If so, please let us know.

Regards,
Vijay N

Re: E810-XXV PFC

vij1 — Mon, 22 Apr 2024 12:18:59 GMT

Hello EvgeniyL,

Greetings for the day!

We have not received a reply from you , and as such, we will be closing your case.

If you want to continue support, please reply to this post and we will reopen your case or create a new one so that we can continue to support you.

Regards,
Vijay N.