Unable to PXE boot guest using SR-IOV VFs

[SKenn2]

Hello,

Config-centric info:

Using HP BL460C-G8 and 560FLB (82599 based) NIC. Host and guest OS is RHEL6.4 (latest - 2.6.32-358.2.1.el6.x86_64). Guest is defined SRIOV interfaces, but when booting, the bios doesn't appear to see any NICs and hence does not pxe boot (seen by examining the BIOS (seabios) output in the console). BTW, we are using libvirt and KVM.

I believe I need to create an optional ROM to load into libvirt, but when trying to build for ipxe, it fails. gpxe which ships with RH (latest - gpxe-roms-qemu-0.9.7-6.9.el6.noarch) has a option ROM (e1000-0x100e.rom), but it doesn't help.

Thanks!

Shawn

========================================================================

# lspci | grep -i virtual

00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05)

01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)

04:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

04:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

04:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

04:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

04:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

04:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

04:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

04:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

# lspci -n -s 04:10.0

04:10.0 0200: 8086:10ed (rev 01)

# lspci -n -s 04:10.1

04:10.1 0200: 8086:10ed (rev 01)

Followed directions here:

http://ipxe.org/howto/romburning http://ipxe.org/howto/romburning

This is the failed build output:

arch/i386/scripts/i386.lds:1: undefined symbol `obj_808610ed' referenced in expression

XML definition for VM:

http://libvirt.org/schemas/domain/qemu/1.0 http://libvirt.org/schemas/domain/qemu/1.0'>

<!--

01-s00c09h0

-->

01-s00c09h0

55ca8e0c-adf9-11e2-b630-e4115b950168

29360128

29360128

7

hvm

destroy

restart

restart

/usr/libexec/qemu-kvm

<!-- KVM virtio channels -->

<...

[Patrick_K_Intel1 · Accepted solution]

Thanx for posting your question.

Always love to see interesting SR-IOV questions!

I will go and double check this with my experts, however I am pretty sure you cannot perform any kind of network boot over a SR-IOV interface. The network boot (PXE, iSCSI Boot & FCoE Boot) all use the option ROM for the booting. A SR-IOV VF does not have access to the option ROM.

Again, I will double check this and get back to you, however I think what you need to do is have an emulated device assigned to the VM that can do the PXE boot and then once the VM OS is running, switch over to SR-IOV.

I'll post back more information when I get it.

thanx,

Patrick

[SKenn2]

I absolutely agree with you that an option ROM is needed. Does Intel offer such a thinking for their devices?? According to libvirt documentation, there is a way to load a option ROM for the device for sriov interfaces. That is the " i listed before. I think al I need is a way to build one.

from libvirt:

rom

The rom element is used to change how a PCI device's ROM is presented to the guest. The optional bar attribute can be set to "on" or "off", and determines whether or not the device's ROM will be visible in the guest's memory map. (In PCI documentation, the "rombar" setting controls the presence of the Base Address Register for the ROM). If no rom bar is specified, the qemu default will be used (older versions of qemu used a default of "off", while newer qemus have a default of "on"). Since 0.9.7 (QEMU and KVM only). The optional file attribute is used to point to a binary file to be presented to the guest as the device's ROM BIOS. This can be useful, for example, to provide a PXE boot ROM for a virtual function of an sr-iov capable ethernet device (which has no boot ROMs for the VFs). Since 0.9.10 (QEMU and KVM only).

[Patrick_K_Intel1]

Intel does have option ROMS to allow the devices to boot via PXE, iSCSI Boot or FCoE Boot. However as noted before, this is not avaialble via a SR-IOV VF.

While libvirt has a mechansim by which you may be able to roll your own for a VF provided you have access to the option ROM binary.

Intel does not provide the option ROM other than attached to the NICs, nor have we done any kind of testing of what libvirt advertises is possible. While I doubt this was the answer you were looking for, I've confirmed my original thesis that you can have an emulated device attached to your VM for PXE boot, then use the VF once the OS is up and running.

thanx,

Patrick

[SKenn2]

Hi Patrick,

You are right. This is NOT the answer I was looking for. While Intel does not support this kind of configuration, I would be most welcome if Intel could provide (without any warranties) the option rom source so I could build my own.

Meanwhile regarding your counter proposal, I do not follow how one would accomplish that unless you are proposing some kind of hot-swap on the ethernet device. Can you elaborate more??

Thanks!!

Shawn

[Patrick_K_Intel1]

Would love to accomidate your wish, however those pesky lawers would probably fire me, even if I knew where to get those ROMs.

It is not uncommon to assign more than one Ethernet interface to a Virtual Machine. You can assign an emulated device that support PXE through the hypervisor that would allow you to get your OS up and running. Then when it is up and running you could, via a script or something disable the emulated device so that only your VF was available for use.

- Patrick

[BRak]

This is actually pretty easy to do. You're on the right track with your entry to libvirt.

So, we're going to compile iPXE to act as the boot firmware. Follow the docs at http://ipxe.org/howto/romburning iPXE - open source boot firmware [howto:romburning] to generate a rom file for your network card (You can compile the rom easily at http://rom-o-matic.eu/ ROM-o-matic.eu | Generate iPXE images | open source network boot firmware , choose 'Advanced', then 'ROM binary (flashable) image (.rom)'). I'm using an i350 chipset, so my PCI ID was 8086 device code 1520. This leaves you with a file 80861520.rom

Modify your libvirt xml to include this line in your :

Destroy and start the VM, and you should see iPXE attempting to network boot. Out of the box, iPXE works just like the standard rom firmware. It has a *ton* of additional features though, so take a look at the documentation!

[SKenn2]

A follow-up to the thread in case anyone is looking....

An option rom is exactly what is needed here, but is unavailable via ipxe project (which what the rom-o-matic uses).

Until it gets pulled in, there is no support.

Thanks,

Shawn