SR-IOV X540-T2 min_tx_rate vf supported?

[AKour]

Is the min_tx_rate supported for vfs created by a X540 nic?

The ip link set command for min_tx_rate does not complain but I see no relevant info at the ip link show results.

If not, what others NICs currently support it?

[Daniel_D_Intel1]

Hello Akour, Thank you for posting in Intel Ethernet Communities. We will check if this command is supported. Please let us know if this is an Intel retail X540-T2 or OEM by providing the markings from the product label. If this is a LOM integrated into a board please inform us. What Linux version are you using? Please provide "ethtool -i <interface>". Let us know if you have any other questions. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[AKour]

Hello Daniel,

It is a retail Intel x540-t2 and is not integrated into the board.

We are using ubuntu 16.04

and the ethtool -i output is:

driver: ixgbe

version: 4.4.6

firmware-version: 0x80000389

expansion-rom-version:

bus-info: 0000:0b:00.0

supports-statistics: yes

supports-test: yes

supports-eeprom-access: yes

supports-register-dump: yes

supports-priv-flags: yes

which Intel cards support min_tx_rate?

Thank you!

[Daniel_D_Intel1]

Hello Akour, Thank you for the reply. We will check if the X540-T2 or other adapters have support for Min_tx_rate. We will provide an update as soon as possible. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[AKour]

I rebuilt my configuration to try also if the  Intel Ethernet Controller X710 for 10GbE SFP+ [8086:1572] cards may support the min_tx_rate feature.

However, I discovered that they behave pretty much the same, if not worse. Although in the latest driver version I installed it mentions the min_tx_rate feature.

So an additional question is if X710 also do not support this feature.

Thanks you,

Best Regards

[Daniel_D_Intel1]

Hello Akour, Thank you for the reply. Are you using ixgbe 5.5.3 and VF driver 4.5.2? This may resolve the issue if the VF driver is not up to date with the host ixgbe. Let us know if you have any other questions. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[Daniel_D_Intel1]

Hello Akour, Please let us know if updating the VF driver allowed the min_tx_rate command to successfully apply. If you have any other questions please let us know. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[AKour]

Hello Daniel D,

I updated my drivers to the latest version for both ixgbe and ixgbevf.

But still no enforcement on the min_tx_rate option. Also the ip link show command does to show any change on the vfs after the ip link set command.

I guess neither the X540-T2 and the X710 do not support the min_tx_rate option over SR-IOV.

[Daniel_D_Intel1]

Hello Akour, Thank you for the reply. The X540-T2 should support the feature, but allow us to double check this along with the X710 series. We will update you soon. Let us know if you have any other questions. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[AKour]

Hello Daniel D,

Let me do a detailed overview of my setup, in order to clarify some points and provide more info of what we try to achieve.

We have 1 Openstack Server with the X540-T2 NIC, and using the SR-IOV agent we bind 2 VFs to 2 VMs. The ixgbe driver is updated on the host and the ixgbevf in the guest, both to the latest versions.

On the other side we have 1 server with a X540-T2 NIC too.

1. We run the pktgen app in the both VMs in order to generate the maximum rate of traffic.
2. When only 1 VM runs it generates about 9.5Gbps of traffic.
3. Then we run the pktgen on the second VM, both VMs send about 5Gbps of traffic each (the VF resources are equally shared, makes sense)
4. We go to the host and on one VF we apply the command "ip link set ens785f1 vf 0 min_tx_rate 6000" for example, to enforce the minimum bandwidth rule.
5. But the VF resources continue to be split equally (5-5), when we should have had 6Gbps of traffic for the first VM and about 4Gbps for the second one.
6. Also I do not see any changes on the state of the interface when I run the ip link show command.

Maybe something in our methodology is wrong, or what we try to achieve is not in any case possible.

Also I failed to find any guide, or example for the min_tx_rate. Only this link which is a mellanox one: https://community.mellanox.com/s/article/howto-configure-rate-limit-per-vf-for-connectx-4-connectx-5

Is there a document with relevant info for the min_tx_rate function?

Best Regards,

[Daniel_D_Intel1]

Hello Akour, Thank you for the information. We will take this into account. We will check if there is any other relevant documentation. Please provide a dmesg output after "modprobe ixgbe". Also provide kern.log in a compressed ZIP. Let us know if you have any questions. Best regards, Daniel D Intel Customer Support Under contract to Intel Corporation

[AKour]

Hello Daniel D,

Thank you for your support and time

Below you can find the dmesg log when I boot up the VMs and apply the min_tx_rate rule

 ixgbe 0000:0b:00.0: setting MAC 66:94:db:bf:de:3d on VF 3

[161061.490342] ixgbe 0000:0b:00.0: Reload the VF driver to make this change effective.

[161061.517521] ixgbevf: eth3: ixgbevf_remove: Remove complete

[161061.517628] ixgbevf 0000:0b:10.6: enabling device (0000 -> 0002)

[161061.581468] ixgbe 0000:0b:00.0 ens2f0: VF Reset msg received from vf 3

[161061.597943] Warning: no configuration for board #24

[161061.597946] Using defaults for all values

[161061.598569] ixgbevf 0000:0b:10.6: Multiqueue Enabled: Rx Queue count = 2, Tx Queue count = 2

[161061.599011] ixgbevf: eth2: ixgbevf_probe: Intel(R) X540 Virtual Function

[161061.599013] 66:94:db:bf:de:3d

[161061.599017] ixgbevf: eth2: ixgbevf_probe: GRO is enabled

[161061.599019] ixgbevf: eth2: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver

[161061.601092] ixgbevf 0000:0b:10.6 enp11s16f6: renamed from eth2

[161120.720572] audit: type=1400 audit(1548934461.697:32): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-3f65f612-5f81-40e3-baa1-b119203424b7" pid=62100 comm="apparmor_parser"

[161120.755423] ixgbevf: enp11s16f6: ixgbevf_remove: Remove complete

[161120.756178] ixgbe 0000:0b:00.0: setting MAC fa:16:3e:f0:97:cc on VF 3

[161120.756181] ixgbe 0000:0b:00.0: Reload the VF driver to make this change effective.

[161120.756235] ixgbe 0000:0b:00.0: Setting VLAN 205, QOS 0x0 on VF 3

[161121.021384] audit: type=1400 audit(1548934461.997:33): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-3f65f612-5f81-40e3-baa1-b119203424b7" pid=62114 comm="apparmor_parser"

[161121.253669] audit: type=1400 audit(1548934462.229:34): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-3f65f612-5f81-40e3-baa1-b119203424b7" pid=62117 comm="apparmor_parser"

[161121.490523] audit: type=1400 audit(1548934462.465:35): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-3f65f612-5f81-40e3-baa1-b119203424b7" pid=62120 comm="apparmor_parser"

[161121.510201] device tap0b0a29c6-5e entered promiscuous mode

[161121.760993] audit: type=1400 audit(1548934462.737:36): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-3f65f612-5f81-40e3-baa1-b119203424b7" pid=62167 comm="apparmor_parser"

[161122.692335] vfio-pci 0000:0b:10.6: enabling device (0000 -> 0002)

[161129.132340] kvm: zapping shadow pages for mmio generation wraparound

[161129.136293] kvm: zapping shadow pages for mmio generation wraparound

[161129.626893] kvm [62173]: vcpu0 unhandled rdmsr: 0x34

[161129.627179] kvm [62173]: vcpu0 unhandled rdmsr: 0x606

[161130.204556] ixgbe 0000:0b:00.0 ens2f0: VF Reset msg received from vf 3

[161336.753699] audit: type=1400 audit(1548934677.724:37): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-edb3eab0-e778-4a51-be77-aa102eb8c10b" pid=63953 comm="apparmor_parser"

[161336.798260] ixgbevf: enp11s16f4: ixgbevf_remove: Remove complete

[161336.798941] ixgbe 0000:0b:00.0: setting MAC fa:16:3e:71:f0:77 on VF 2

[161336.798944] ixgbe 0000:0b:00.0: Reload the VF driver to make this change effective.

[161336.798962] ixgbe 0000:0b:00.0: Setting VLAN 205, QOS 0x0 on VF 2

[161337.025877] audit: type=1400 audit(1548934677.996:38): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-edb3eab0-e778-4a51-be77-aa102eb8c10b" pid=63972 comm="apparmor_parser"

[161337.253207] audit: type=1400 audit(1548934678.224:39): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-edb3eab0-e778-4a51-be77-aa102eb8c10b" pid=63975 comm="apparmor_parser"

[161337.667732] audit: type=1400 audit(1548934678.640:40): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-edb3eab0-e778-4a51-be77-aa102eb8c10b" pid=63978 comm="apparmor_parser"

[161337.686439] device tapefd88538-7c entered promiscuous mode

[161337.919668] audit: type=1400 audit(1548934678.892:41): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-edb3eab0-e778-4a51-be77-aa102eb8c10b" pid=64022 comm="apparmor_parser"

[161355.655544] vfio-pci 0000:0b:10.4: enabling device (0000 -> 0002)

[161364.400166] kvm: zapping shadow pages for mmio generation wraparound

[161364.407013] kvm: zapping shadow pages for mmio generation wraparound

[161364.946917] kvm [64026]: vcpu1 unhandled rdmsr: 0x34

[161364.947229] kvm [64026]: vcpu1 unhandled rdmsr: 0x606

[161365.519296] ixgbe 0000:0b:00.0 ens2f0: VF Reset msg received from vf 2

And here is the kern.log

https://www.dropbox.com/s/syyo4tfy87x4fms/kernlog-akourt.tar.gz?dl=0

BR,

Akis Kourtis

[VincentT_Intel]

Hi Akis, Thank you for providing the Dmesg output and kern.log. We'll check them and update this thread as soon as possible. Best Regards, Vince T. Intel Customer Support Under Contract to Intel Corporation

[VincentT_Intel]

Hi Akis, Based from the Dmesg output and kern.log, we've noticed the X540 was installed in a x4 PCI slot, please try to move it to a x8 slot. The current x4 PCI slot terminates in a PCH PCI Express Root Complex that does not support ARI. Desired number of VFs may not get instantiated because of the lack of ARI support. Another suggestion is to use SysFS instead of max_vfs. On newer Linux Kernel such as the one you're using. max_vfs module parameter has been deprecated. You can use the following command to create VFs. echo '7' > /sys/bus/pci/devices/0000:03:00.0/sriov_numvfs Ixgbevf driver is being loaded on the host when the VFs are created. These VFs may not be available for assignment to the VM because they are being claimed by the VF driver in the host. One method to get around this is to blacklist the VF driver in the host Linux server. Once the VF driver is blacklisted it will not load in the Linux host and VF can be assigned to the VM. For sample steps on how to blacklist the VF, please refer to page 14 of this guide - https://www.intel.sg/content/www/xa/en/embedded/products/networking/xl710-sr-iov-config-guide-gbe-linux-brief.html It also shows the ixgbevf driver version is 3.3.2, please update the ixgbevf driver to version 4.5.2 and retest. Latest ixgbevf diver can be downloaded from https://sourceforge.net/projects/e1000/files/ixgbevf%20stable/4.5.2/ Best Regards, Vince T. Intel Customer Support Under Contract to Intel Corporation

[AKour]

Hello Vince T. ,

The X540 was moved to a x4 slot during our try with the X710, we then moved it back to the x8 and could again spawn VFs.

When the X540 was at the x4 I could not even run the echo command. So the log contains this transfer. In the latest events you will see I spawn VFs.

Regarding the ixgbevf, I install the latest version inside the VM, as it is where it is used.

As to the ixgbevf claimed by the host point you raise, when I boot the VMs I can see the ixgbevf driver loaded inside the VM, and the virtual port on the host is claimed by SR-IOV, otherwise the VM would not even boot.

(I dealt with this issue when I had not blacklisted the vf drivers).

I am really concerned about the fact that there is no guide to explain the steps and how to do them in an Intel card (always talking about min_tx_rate).

Also it is not displayed in the ip link show output, whereas the max_tx_rate is.

All these facts lead me to the fact that maybe it is not yet fully supported. Do you have a proof or something that it works on either x540 or x710?

Or some guidelines how to achieve, maybe the setup with all the openstack and stuff blocks something.

Anyway thanks for your support.

BR,

Akis

[VincentT_Intel]

Hi Akis, Thank you for your response. Let me get back to you regarding guidelines to setup the min_tx_rate in x540 \ x710. Best Regards, Vince T. Intel Customer Support Under Contract to Intel Corporation

[VincentT_Intel · Accepted solution]

Hi Akis, After further confirmation, we don't have adapter that support min_tx_rate. You may consider using the rate function to limit the maximum tx rate on the VF. Kindly refer to sample configuration below: ip link set eth0 vf 0 rate 1000 (This command sets TX Rate Limit of 1000Mbps for VF 0) We sincerely apologize for the confusion this might have caused. Please let us know if there's other assistance needed. Best Regards, Vince T. Intel Customer Support Under Contract to Intel Corporation

[VincentT_Intel]

Hi Akis, Hope we were able to answer your question. Please let us know if there's other assistance needed. Best Regards, Vince T. Intel Customer Support Under Contract to Intel Corporation