Rate limiting on Virtual Function

[srang3]

Hi,

I am using Intel 10 Gig Dual-Port Ethernet PCI Card (BN8110470) ixgbe driver on Ubuntu 14.04 Host.

I was able to create VF's which are listed below from lspci command:

02:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

02:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

03:10.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:10.1 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:10.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:10.3 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:10.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:10.5 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:10.6 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:10.7 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:11.0 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:11.1 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:11.2 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:11.3 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:11.4 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

03:11.5 Ethernet controller: Intel Corporation X540 Ethernet Controller Virtual Function (rev 01)

The VF's are also listed from ip link command:

20: rename20: mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

link/ether a0:36:9f:11:b8:10 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

28: rename28: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether a0:36:9f:11:b8:12 brd ff:ff:ff:ff:ff:ff

vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

vf 6 MAC 00:00:00:00:00:00, vlan 10, spoof checking on, link-state auto

As visible from dev rename28, vf 6, assigning vlan is working as expected,

ip link set rename28 vf 6 vlan 10

But setting rate limit on same VF is working,

sudo ip link set rename28 vf 6 rate 250

RTNETLINK answers: Invalid argument

The guide used to realize this configuration is here, http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html Configure QoS with Intel® Flexible Port Partitioning

I know that I should be using the latest iproute2, I am using iproute2-ss131122 which is never version.

[VincentT_T_Intel]

Hi Srangare, just checking if you're using the VF driver posted in the downloadcenter, sharing the link below:

82599 or X540 - http://downloadcenter.intel.com/download/18700/Network-Adapter-Virtual-Function-Driver-for-10-Gigabit-Network-Connections http://downloadcenter.intel.com/download/18700/Network-Adapter-Virtual-Function-Driver-for-10-Gigabit-Network-Connections

82576 or i350 - http://downloadcenter.intel.com/download/18298/Network-Adapter-Virtual-Function-Driver-for-Gigabit-Network-Connections-for-Linux- http://downloadcenter.intel.com/download/18298/Network-Adapter-Virtual-Function-Driver-for-Gigabit-Network-Connections-for-Linux-

regards,

vince

[TKnau]

Hi Vince,

we have a similar problem. lspci reports our card as

03:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

We use the following drivers on the machine

thomas@sep1:~$ modinfo ixgbe

filename: /lib/modules/3.16.0-36-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko

version: 3.23.2.1

thomas@sep1:~$ modinfo ixgbevf

filename: /lib/modules/3.16.0-36-generic/kernel/drivers/net/ixgbevf/ixgbevf.ko

version: 2.16.1

Both of which I believe to be the most recent version downloaded from Intel's website.

Setting a VLAN with

thomas@sep1:~$ sudo ip link set em1 vf 0 vlan 0

works. Setting a rate limit with

thomas@sep1:~$ sudo ip link set em1 vf 0 rate 500

RTNETLINK answers: Invalid argument

does not. We are a bit clueless how to proceed from here.

Thanks for you time,

Thomas.

[VincentT_T_Intel]

Hi Thomas, let me further check this one.

thanks,

Vince

[TKnau]

We figured out the "problem" by instrumenting the kernel driver with printf()s. It seems the adapter only allows rate limiting if it is in 10Gbps mode.

I'm curious what is the reason for that?

We will test again, once we have an appropriate 10Gbps cable.

[VincentT_T_Intel]

Hi Thomas, glad to know it works, currently checking on your inquiry. will update this thread as soon as possible.

regards,

Vince

[VincentT_T_Intel]

Hi Thomas, based from the internal testing - using Ubuntu 14.04 LTS with ixgbe driver 3.15.1-k, ixgbevf version 2.11.3-k and same iproute2 utility. rate limit was successful on 1Gbps and 10Gbps. Please share the Ubuntu version used if it's 14.04 or 14.04.1, 14.04.2 and are you using stock kernel or custom modified kernel. thanks

regards,

Vince

[TKnau]

Not that it really matters anymore, but we are using ixgbevf 2.16.1 and ixgbe 3.23.2.1 on Ubuntu 14.10 (kernel 3.16.0-36-generic).

If I look at the ixgbe code, specifically in ixgbe_srio.c for function ixgbe_ndo_set_vf_bw(), I can see that it tests for 10 Gbps mode before allowing me to set the rate limit. We've instrumented the code, i.e. added printfs, to see that this is actually the place where setting the rate limit fails.

Now that we have the adapter in 10 Gbps mode, setting the rate limit works. Previously we only had a CAT5 cable, which is why the adapter was only operating in 1Gbps mode.

[st4]

Hi Thomas,

Thank you for the reply. We will check on this.

rgds,

wb

[VincentT_T_Intel]

Hi Thomas, this is to confirm your findings is accurate about rate limit would test the link on 10Gbps

regards,

Vince

[RB_]

Hi,

I have the similar problem on a 1 Gbps link.

Do you confirm that rate limit is only working on 10 Gbps link ?

Thanks for your help

Ronan