E810-XXV PFC

[EvgeniyL]

I have server with Intel(R) Ethernet Controller E810-XXV for SFP.

Server working on VMware 7.0.3. I want use RDMA for vSAN, but i see failed configuratiob - The PFC is not enabled. 

Now
Mode: 3 - IEEE Mode
PFC Enabled: false

I want set PFC Enabled in true. How to set PFC Enabled values to true?

[vij1]

Hello Evgeniyl,

Greetings for the day!

For using RDMA, download the irdman driver from the VMware site. After loading the RDMA driver(irdman), the NIC will be automatically loaded in ROCEv2 mode.

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io

Refer to "Driver setup on Host" section in irdman driver release notes for detailed explanation on how switching between ROCEv2 and iWARP happens.

Regards,

Vijay N.

[EvgeniyL]

Hi Vijay,

i use latest driver for icen and irdman:

esxcli software vib list |grep icen
icen 1.12.5.0-1OEM.702.0.0.17630552 INT VMwareCertified 2024-02-08

esxcli software vib list |grep ird
irdman 1.4.3.0-1OEM.700.1.0.15843807 INT VMwareCertified 2024-02-08

This is my model my card

VMware Compatibility Guide - I/O Device Search

recommended drivers

ESXi 7.0 U3 icen version 1.12.5.0-1OEM
ESXi 7.0 U3 irdman version 1.4.3.0

When i check PFC, PFC

esxcli network nic dcb status get -n vmnic5
Nic Name: vmnic5
Mode: 3 - IEEE Mode
Enabled: true
Capabilities:
Priority Group: true
Priority Flow Control: true
PG Traffic Classes: 8
PFC Traffic Classes: 8
PFC Enabled: false
PFC Configuration: 0 0 0 0 0 0 0 0
IEEE ETS Configuration:
Willing Bit In ETS Config TLV: 0
Supported Capacity: 8
Credit Based Shaper ETS Algorithm Supported: 0x0
TX Bandwidth Per TC: 0 0 0 0 0 0 0 0
RX Bandwidth Per TC: 0 0 0 0 0 0 0 0
TSA Assignment Table Per TC: 0 0 0 0 0 0 0 0
Priority Assignment Per TC: 0 0 0 0 0 0 0 0
Recommended TC Bandwidth Per TC: 0 0 0 0 0 0 0 0
Recommended TSA Assignment Per TC: 0 0 0 0 0 0 0 0
Recommended Priority Assignment Per TC: 0 0 0 0 0 0 0 0
IEEE PFC Configuration:
Number Of Traffic Classes: 8
PFC Configuration: 0 0 0 0 0 0 0 0
Macsec Bypass Capability Is Enabled: 0
Round Trip Propagation Delay Of Link: 0
Sent PFC Frames: 0 0 0 0 0 0 0 0
Received PFC Frames: 0 0 0 0 0 0 0 0
DCB Apps:
App Type: L2 Ethertype
Protocol ID: 0x8906
User Priority: 0x3

App Type: L2 Ethertype
Protocol ID: 0x0
User Priority: 0x0

[Meghak]

Hello Evgeniyl,

Greetings for the day!

Thank you for confirming the details.

Please allow us some time to review the details shared and we will get back with an update as soon as possible.

Regards,

Megha K

[vij1]

Hello Evgeniyl,

Good day!

Please find below the link for enabling PFC:

Intel® Ethernet 800 Series Linux Flow Control Configuration Guide for RDMA Use Cases:

[Link to Example 1: 800 Series Back-to-Back PCP PFC with Single TC](https://edc.intel.com/content/www/us/en/design/products/ethernet/800-series-linux-flow-control-configuration-guide-for-rdma-use-c/example-1-800-series-800-series-back-to-back-pcp-pfc-with-single-tc/)

[Link to Option A: PCP PFC Set Priorities for Drop or No Drop](https://edc.intel.com/content/www/us/en/design/products/ethernet/800-series-linux-flow-control-configuration-guide-for-rdma-use-c/option-a-pcp-pfc-set-priorities-for-drop-or-no-drop/)

Best regards,

Vijay N.

[EvgeniyL]

Hi Vijay,

Thanks, i saw 1 link early but this both link not for VMware. My problem with VMware and this card. How enabling PFC on ESXi ?

[vij1]

Hello Evgeniyl,

Greetings for the day!

Thank you for sharing the datails. We are checking on this issue and will get back as soon as possible.

Regards,

Vijay N.

[vij1]

Hello Evgeniyl,

Greetings for the day!

Thank you for sharing the datails. We are checking on this issue and will get back as soon as possible.

Regards,

Vijay N.

[vij1]

Hello Evgeniyl,

Greetings for the day!

DCB configuration is part of the DCB supported switch to which E810 NIC is connected to and is not directly performed in ESXi host.

Note that DCB configuration in Intel NIC is done though DCBX protocol running in SW (dcbx agent that is part ESXi). Where, there will be message exchanges between the link partners to converge on the configuration. Intel NIC is in willing mode, meaning it will accept the configuration coming from the link partner. So, all the DCB configuration needs to be done in the DCB supported switch (link partner) to get it synchronized with E810 NIC.

Regards,

Vijay N.

[EvgeniyL]

Hi Vijay,

We use in our infrastructure switch CISCO N9K with config:

lldp dcbx version ieee
  switchport mode trunk
  switchport trunk allowed vlan 1- 1024
  priority-flow-control mode on
  priority-flow-control watch-dog-interval on
  spanning-tree port type edge trunk
  mtu 9216
  logging event port link-status
  logging event port trunk-status
  service-policy type qos input ROCE_POLICY
  service-policy type queuing output ROCE_QUEUING_OUT
  no shutdown

i create cluster on E810 NIC  and have warning

i have host on this switch  with network card Mellanox Technologies MT27800 Family [ConnectX-5]  and i havent problem with PFC 

Problem only with  Intel(R) Ethernet Controller E810-XXV for SFP !

[vij1]

Hello Evgeniyl,

Greetings for the day!

Thank you for sharing the datails. We are checking on this issue and will get back as soon as possible.

Regards,

Vijay N.

[Poojitha]

Hi EvgeniyL,

Good day!

We would like to inform you that the method to manually configure the DCB PFC using driver-specific parameters. To use this method, please check the vendor-specific driver documentation.

For example, in the icen driver, you can set the PFC priority value to 3 by running the following command and rebooting the host:

#esxcli system module parameters set -m icen -p "pfctx=0x08, pfcrx=0x08"

esxcli system module parameters set: -m icen -p "pcp_force=3"

Once you run the above commands, reboot the system. 

 Also, we recommend you check the switch configuration:

dcb priority-flow-control enable

dcb priority-flow-control priority 3: enable

interface ethernet 1/1-1/6 dcb priority-flow-control mode on force

By default, all the configuration for DCP on E810 is acquired from the switch configuration. 

Regards,

Poojitha

[vij1]

Hello EvgeniyL,

Greetings for the day!

We hope this message finds you well. We are sending a follow up if you have any questions from the previous post. If so, please let us know.

Regards,
Vijay N

[vij1]

Hello EvgeniyL,

Greetings for the day!

We have not received a reply from you , and as such, we will be closing your case.

If you want to continue support, please reply to this post and we will reopen your case or create a new one so that we can continue to support you.

Regards,
Vijay N.