Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4811 Discussions

SR-IOV vs nPAR

KEgef
Beginner
6,190 Views

Hi,

For a XEN environment I have doubts if SR-IOV will provide the required features compared to nPAR.

Servers will be Dell M620 with either Intel X520 or Broadcom 57711, running Linux kernel 3.8, XEN 4.2 and openvswitch 1.7

I need separate networks on the hypervisor:

- Management

- Storage

- Openvswitch

- Passthrough for VMs

So my questions are:

- Can I use the PF's on the hypervisor?

- Can I use 4 VFs on the hypervisor and the rest for VM passthrough?

- Can I bond VFs with LACP?

- Can I use a VF as a iSCSI HBA?

- Is DCB supported in above configuration?

- Can I guarantee 1 VF 50% and let the rest use the default round robin fairness?

Hope someone is able to help me with answers or documentation, as I have had difficulties finding it.

Thanks.

Regards

Kristoffer

0 Kudos
16 Replies
Patrick_K_Intel1
Employee
4,371 Views

Thanx for posting to our site.

SR-IOV and nPAR are technologies that provide similar capabilities, however they do so in different ways. SR-IOV is of course a PCI SIG standard, while nPAR is specific to a Server OEM both have their strong and weak points.

Intel Ethernet devices, including the X520 currently do not support nPAR. The Intel SR-IOV solution is pretty full-featured however and does support most of what you list below.

We have quite a bit of documentation available that discusses much of what you have listed below, they are documented in this blog: /community/wired/blog/2012/08/10/all-you-ever-wanted-to-know-about-intel-sr-iov-and-flexible-port-partitioning http://communities.intel.com/community/wired/blog/2012/08/10/all-you-ever-wanted-to-know-about-intel-sr-iov-and-flexible-port-partitioning

Answers to your specific questions:

Can I use the PF's on the hypervisor? [Patrick] Yes, absolutely

 

Can I use 4 VFs on the hypervisor and the rest for VM passthrough? [Patrick] Yes, however only on Linux/KVM

 

Can I bond VFs with LACP? [Patrick] Yes – I have a paper listed on blog post above specifically on bonding

Can I use a VF as a iSCSI HBA? [Patrick] Yes, however iSCSI over DCB is not supported

 

Is DCB supported in above configuration? [Patrick] Yes, DCB is supported in this configuration. As long as DBC is configured for Hypervisor use and not in VMs. It's only supportedin Linux operating systems and will need kernel upgrade and Intel 10GB driver upgrade.

Can I guarantee 1 VF 50% and let the rest use the default round robin fairness?[Patrick] remember that rate limiting is for Tx side only, in order to limit Rx, you must have a switch limit the bandwidth. Please see the papers and videos listed on the above blog for 'FPP', should give you aclear picture of the technology.

Again, thanx for visiting our site and asking questions!

- Patrick

KEgef
Beginner
4,372 Views

Hi Patrick,

Thanks for your answers and links - I read through it and a few things are still unclear.

Can I use the PF's on the hypervisor? [Patrick] Yes, absolutely

So do I understand it correctly that two 10G interfaces with 2 VFs each will look like:

eth0: PF

eth1: VF

eth2: VF

eth3: PF

eth4: VF

eth5: VF

Can I use 4 VFs on the hypervisor and the rest for VM passthrough? [Patrick] Yes, however only on Linux/KVM

You don't mention XEN - is there a reason that only KVM is supported?

As I understand it, I can hide certain VFs from the hypervisor and use them on the VMs?

In the configuration examples I can find, ixgbevf is blacklisted in the hypervisor, which make me think that the hypervisor cannot use the VFs? Do you know if this is the issue with XEN?

Can I bond VFs with LACP? [Patrick] Yes – I have a paper listed on blog post above specifically on bonding

Reading http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html it seems that LACP is not working due to some antispoofing issue - do you know if this is solved?

Can I use a VF as a iSCSI HBA? [Patrick] Yes, however iSCSI over DCB is not supported

Does this mean that I need to run iSCSI on the PF if I want to use DCB, or is DCB not supported at all?

Is DCB supported in above configuration? [Patrick] Yes, DCB is supported in this configuration. As long as DBC is configured for Hypervisor use and not in VMs. It's only supportedin Linux operating systems and will need kernel upgrade and Intel 10GB driver upgrade.

Now I'm confused ;-) So DCB is supported on a VF if a use it on the hypervisor?

Can I guarantee 1 VF 50% and let the rest use the default round robin fairness?[Patrick] remember that rate limiting is for Tx side only, in order to limit Rx, you must have a switch limit the bandwidth. Please see the papers and videos listed on the above blog for 'FPP', should give you aclear picture of the technology.

Reading http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/config-qos-with-flexible-port-partitioning.html it appears that I can only limit bandwidth on interfaces, not guarentee a specific interface a certain amount of bandwidth, unless limiting all the other interfaces?

As I understand it nPAR can provide bandwidth guarentees pr interface - does SR-IOV not have this option?

Would DCB handle this?

Thanks

Regards

Kristoffer

0 Kudos
Patrick_K_Intel1
Employee
4,372 Views

Kristoffer - I haven't forgotten about you. Still digging into your questions.

- Patrick

0 Kudos
KEgef
Beginner
4,371 Views

Great - thanks Patrick.

0 Kudos
Patrick_K_Intel1
Employee
4,371 Views

Kristoffer,

Let's see if I can answer all your questions one by one.

To be clear, there is a difference between Xen and XenServer. You can do everything on Xen that you can do on KVM - although it is done differently for some things. XenServer is a flavor of Xen that doesn't support all these goodies. For example iSCSI is not support by XenServer, but is in OpenSource Xen.

You don't want to blacklist ixgbevf, because that will (as you found out) not load the VF driver within the hyervisor. We have a whitepaper that discusses how to setup VF's for both use with a VM and within the hypervisor that should help you out: http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-suse-linux-xen-tech-brief.html http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-suse-linux-xen-tech-brief.html

The problem with LACP has been fixed in the latest release of the iproute2 utility (that has not yet made it into the distros from what I understand). You need to update the utility itself, along with the driver and the kernel. Use the spoofchk parameter to disable anti-spoofing.

Regarding DCB - DCB is not supported in a VF, only in the PF.

Lastly, on the topic of bandwidth guarantees. As you surmised, you can only guarantee a VF to have a guaranteed bandwidth by rate limiting all other VF's. This is true no matter what kind of partitioning technology you choose. Flex-10, NPAR or SR-IOV all have to limit the other partitions to ensure bandwidth reservations. There is only 10G to go around, so if you want to guarantee say 5Gbps to one partition (VF, NPAR, Flex-10 ports) then all the other partitions must never exceed a combined 5Gbps limit and therefore must be limited.

You can do a more flexible type of rate limiting using some kind of chron job or daemon if you wanted to get fancy, where you monitor all the other VF's and if the combined bandwidh exceeds, or nears the 5Gbps AND the partition you want to have guaranteed BW for is using 5Gbps then you can rate limit the other VF's. This is just an idea, I've not actually done it in a daemon or anything, just manually tweaking with the gui I created for the demo videos.

Hope this helps, and thanx for posting!

- Patrick

0 Kudos
KEgef
Beginner
4,372 Views

Thanks Patrick, then only the QOS problem remains.

This is from Qlogic nPAR documentation, stating that bandwidth guarantee is supported and is dynamic across the interfaces. Or am I misunderstanding this?

 

Q: Is one virtual port's unused bandwidth available for use by other active virtual ports?

Yes. The minimum settings are bandwidth guarantees, specified as a percentage of the link speed. If one or more virtual ports aren't consuming their full allotment, that bandwidth can be temporarily consumed by other virtual ports if they need more than their guaranteed allotment.

 

Also, if the VF is hidden from the hypervisor, I don't see how I can rate limit the traffic outside the VM, which makes it very difficult to guarantee bandwidth to a interface in the hypervisor for storage etc.

Is there a way to rate limit from the hypervisor?

Thanks for your answers!

0 Kudos
Patrick_K_Intel1
Employee
4,371 Views

Kristoffer,

Thanx for your great questions!

We do not have a mechansim to automagically do as the Qlogic nPAR documentation claims. All these technologies have their quirks and issues. With the Intel SR-IOV solution you can do as you are asking, however you would need to do so with a daemon or something like that you wrote yourself.

In regards to being able to manipulate the VF from the hypervisor even though the VF itself is not available to the hypervisor - this is true, however the way you control the VF from the hypervisor is via the PF that owns the VF. So you specify the PF and then the VF # in order to say rate limit, or assign a VLAN etc.

regards,

Patrick

0 Kudos
idata
Employee
4,371 Views

Hi Patrick,

You mentioned here that LACP should be supported with SR-IOV, however, I was unable to get it configured properly. I'm running Ubuntu 12.04 64-bit server with latest patches (kernel 3.2.0-39, ixgbe-3.13.10 and iproute_20121211 - Debian release). I'm trying to configure a bond with 2 interfaces in LACP bundle and that works perfectly without SR-IOV and when I bridge virtual machine to this bond. When I turn on SR-IOV, I'm still able to use this bond normally until I get a VM started (KVM QEMU 1.0, libvirt 0.9.8) and try to configure LACP inside it. At that point, connectivity just gets totally broken, like in typical scenario when server and router don't have same LACP configuration set. I've also turned off spoof checking through ip utility. Is there something so obvious what I'm missing?

Could you please explain how did you make it to work, which versions did you use, etc.?

Thanks in advance!

Tomislav

0 Kudos
Patrick_K_Intel1
Employee
4,372 Views

Tomislav,

Please explain what all is in your bridge, and where the bridge is - is it in the VM or the hypervisor. And what all is in the bridge (what interfaces).

thanx,

Patrick

0 Kudos
idata
Employee
4,372 Views

I have a router configured LAG 802.3ad aggregation on those 2 ports (eth4 and eth7) on the server, a part from /etc/network/interfaces on Ubuntu:

auto eth7

iface eth7 inet manual

bond-master bond1

auto eth4

iface eth4 inet manual

bond-master bond1

auto bond1

iface bond1 inet manual

bond-mode 802.3ad

bond-miimon 100

bond-lacp-rate 1

bond-slaves none

Additionally I created a bridge br1 which has bond1 attached to itself and ports from virtual machines (hypervisor is KVM and ports are simply bridged). This setup works well without SR-IOV. It also works fine when I enable SR-IOV, but before I attach virtual machine to VFs.

When I attach a virtual machine to VFs (Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)) of eth4 and eth7 and configure the same bond in the VM, the whole connectivity (to host and to the VMs) is broken. When I don't configure LACP (bond1 and br1) on the host and use VFs in the virtual machine, the connectivity towards VM is broken again. And I've doublechecked the setting for spoof checking - it's off:

eth7:

...

vf 14 MAC 96:9e:b8:06:0c:2c, spoof checking off

Thanks!

T.

0 Kudos
KEgef
Beginner
4,372 Views

Thanks Patrick,

In case of problems with LACP over VFs, can I just use LACP over the PFs, while using the VFs for other tasks?

I'm having difficulties understanding how LACP could work over multiple pairs of VFs at the same time - do you know if this is possible?

Regards

Kristoffer

0 Kudos
Patrick_K_Intel1
Employee
4,372 Views

Kristoffer,

Sorry for the delay - took a few days off.

In the /community/wired/blog/2012/06/25/latest-flexible-port-partitioning-paper-is-now-available-learn-about-qos-and-sr-iov paper written on this topic, I believe we recommend not doing teaming at the PF level. For load balancing, you will definately want to load balance between VF's from different PF's (or even different physical NICs if you like). Load balancing between VF's from the same PF won't do you any good, because it would end up using the same physical port (PF).

Does that help?

- Patrick

0 Kudos
KEgef
Beginner
4,372 Views

Hi Patrick,

OK - teaming/bonding must be on VFs.

Also I understand that bonds need VFs from different PFs to work.

What I don't understand is if it's possible to have 2 LACP bonds on the same PFs, like this:

PFs are eth0 and eth3

eth1+eth4: LACP

eth2+eth5: LACP

Is this doable ?

Thanks - this will be the last piece of info for now ;-)

Regards

Kristoffer

0 Kudos
Patrick_K_Intel1
Employee
4,372 Views

Hi Kristoffer,

You can assign two VF's from the same PF to a VM and team they anyway you wish, nothing to prevent you from that. However you probably won't gain anything.

You would be load balancing from the same physical port, so there wouldn't really be any balancing occuring, it's all coming and going to the same cable.

I hope this helps.

- Patrick

0 Kudos
KEgef
Beginner
4,371 Views

I don't want to assign VFs from the same PF in the team - I want to assign 2 x 2 VFs from different PFs in two teams - like this:

PF1: eth0

PF1(VF1): eth1 (LACP team 1)

PF1(VF2): eth2 (LACP team 2)

PF2: eth3

PF2(VF1): eth4 (LACP team 1)

PF2(VF2): eth5 (LACP team 2)

Two lacp teams assigned to two separate PFs.

What I have doubts about is, if the switch can handle multiple LACP teams on the same ports?

Makes sense?

Thanks

0 Kudos
Patrick_K_Intel1
Employee
4,372 Views

I think I have it now - thanx for clarifying.

There is no limitation in the VF's for this, however you are likely correct that the switches would not support more than LACP bond - however I am not a switch expert so while I find it doubtful it would work, I can't say with 100% certainty.

- Patrick

0 Kudos
Reply