82599 using SRIOV in KVM

[idata]

Hello:

I get stuck in here. Last time I post a disscussion about SR-IOV using of 82599 in XEN. I get reply from some of you, but I still failed to succeed in making it work. Now I read another article succeed in make it working in KVM written by Waseem.

Now I do as the article said, but I stucked as well. The problem remained to be " not enough MMIO resources for SR-IOV error ". And I am very sure that I have enable all the settings in BIOS and kernel as well as msi after checking the enviroment again and again.

I will post the relative information of my machine here in the attachment.

io memory:

00000000-0000ffff : reserved

00010000-0009dbff : System RAM0009dc00-0009ffff : reserved000a0000-000bffff : Video RAM area000c0000-000c7fff : Video ROM000c8000-000c8fff : Adapter ROM000c9000-000c9fff : Adapter ROM000e0000-000fffff : reserved 000f0000-000fffff : System ROM00100000-bf78ffff : System RAM 00400000-00812005 : Kernel code 00812006-00a3d007 : Kernel data 00ac5000-00bd18a3 : Kernel bss 02000000-09ffffff : Crash kernelbf790000-bf79dfff : ACPI Tablesbf79e000-bf7cffff : ACPI Non-volatile Storagebf7d0000-bf7dffff : reservedbf7e0000-bf7ebfff : RAM bufferbf7ec000-bfffffff : reservede0000000-efffffff : PCI MMCONFIG 0 [00-ff] e0000000-efffffff : reserved e0000000-efffffff : pnp 00:0ef8cdc000-f8cdffff : 0000:80:16.7 f8cdc000-f8cdffff : ioatdmaf8ce0000-f8ce3fff : 0000:80:16.6 f8ce0000-f8ce3fff : ioatdmaf8ce4000-f8ce7fff : 0000:80:16.5 f8ce4000-f8ce7fff : ioatdmaf8ce8000-f8cebfff : 0000:80:16.4 f8ce8000-f8cebfff : ioatdmaf8cec000-f8ceffff : 0000:80:16.3 f8cec000-f8ceffff : ioatdmaf8cf0000-f8cf3fff : 0000:80:16.2 f8cf0000-f8cf3fff : ioatdmaf8cf4000-f8cf7fff : 0000:80:16.1 f8cf4000-f8cf7fff : ioatdmaf8cf8000-f8cfbfff : 0000:80:16.0 f8cf8000-f8cfbfff : ioatdmaf8cfe000-f8cfffff : pnp 00:11f8d00000-f8ffffff : PCI Bus 0000:85 f8d80000-f8dfffff : 0000:85:00.1 f8e00000-f8e7ffff : 0000:85:00.1 f8e00000-f8e7ffff : ixgbe f8ef8000-f8efbfff : 0000:85:00.1 f8ef8000-f8efbfff : ixgbe f8efc000-f8efffff : 0000:85:00.0 f8efc000-f8efffff : ixgbe f8f00000-f8f7ffff : 0000:85:00.0 f8f80000-f8ffffff : 0000:85:00.0 f8f80000-f8ffffff : ixgbef9000000-f9ffffff : PCI Bus 0000:07 f9000000-f9ffffff : 0000:07:04.0fadd8000-fadd80ff : 0000:00:1f.3fadda000-fadda3ff : 0000:00:1d.7 fadda000-fadda3ff : ehci_hcdfaddc000-faddffff : 0000:00:16.7 faddc000-faddffff : ioatdmafade0000-fade3fff : 0000:00:16.6 fade0000-fade3fff : ioatdmafade4000-fade7fff : 0000:00:16.5 fade4000-fade7fff : ioatdmafade8000-fadebfff : 0000:00:16.4 fade8000-fadebfff : ioatdmafadec000-fadeffff : 0000:00:16.3 fadec000-fadeffff : ioatdmafadf0000-fadf3fff : 0000:00:16.2 fadf0000-fadf3fff : ioatdmafadf4000-fadf7fff : 0000:00:16.1 fadf4000-fadf7fff : ioatdmafadf8000-fadfbfff : 0000:00:16.0 fadf8000-fadfbfff : ioatdmafadfc000-fadfc3ff : 0000:00:1a.7 fadfc000-fadfc3ff : ehci_hcdfae00000-faffffff : PCI Bus 0000:01 fae00000-fae1ffff : 0000:01:00.0 fae20000-fae3ffff : 0000:01:00.0 fae40000-fae5ffff : 0000:01:00.1 fae60000-fae7ffff : 0000:01:00.1 faee0000-faefffff : 0000:01:00.1 faf00000-faf1ffff : 0000:01:00.1 faf00000-faf1ffff : igb faf20000-faf3ffff : 0000:01:00.1 faf20000-faf3ffff : igb faf98000-faf9bfff : 0000:01:00.1 faf98000-faf9bfff : igb faf9c000-faf9ffff : 0000:01:00.0 faf9c000-faf9ffff : igb fafa0000-fafbffff : 0000:01:00.0 fafc0000-fafdffff : 0000:01:00.0 fafc0000-fafdffff : igb fafe0000-faffffff : 0000:01:00.0 fafe0000-faffffff : igbfb000000-fbefffff : PCI Bus 0000:07 fb000000-fb7fffff : 0000:07:04.0 fbefc000-fbefffff : 0000:07:04.0fec00000-fec00fff : IOAPIC 0fec8a000-fec8afff : IOAPIC 1 fec8a000-fec8afff : 0000:00:13.0fec9a000-fec9afff : IOAPIC 2 fec9a000-fec9afff : 0000:80:13.0fed00000-fed003ff : HPET 0fed1c000-fed1ffff : pnp 00:01 fed1c000-fed1ffff : pnp 00:0bfed20000-fed3ffff : pnp 00:0bfed40000-fed8ffff : pnp 00:0bfee00000-fee00fff : Local APIC fee00000-fee00fff : reserved fee00000-fee00fff : pnp 00:0dffc00000-ffffffff : reserved100000000-33fffffff : System RAM

lspci:

00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)

Subsystem: Super Micro Computer Inc Device 0400 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-   Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit- Address: 00000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [90] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us<p>  ExtTag+ RBE+ FLReset- DevCtl: Report err...

[Patrick_K_Intel1]

The problem you are having is either with your BIOS or your OS.

Have you tried using Red Hat 6.1 KVM? I know that some version of Red Hat 5.5 Xen and KVM had a bug in regards to the Intel 82599 10Gb Ethernet Controller.

I don't see where you indicated what platform and BIOS version you are using. Might be useful in letting people help you out.

[idata]

Hello:

You can use key word dmidecode to find the information in what I have posted above. Beside my OS is redhat 6.0.

[Patrick_K_Intel1]

What is the actual Server you are using? What model # , BIOS revision etc.

I'll see what I can find out for you - as I've mentioned before, it is likely a BIOS issue.

[idata]

this is old but I will post this information here so people will at least have a chance of seeing this.

1. first disable selinux

2. rmmod kvm_intel

3. rmmod kvm

4. modinfo kvm

5. there will be one parameter listed there

6. modprobe kvm allow_unsafe_assignable_interrupts=1

7. modprobe kvm_intel

this may work.

selinux can in some instances block the assignment, since it doesnt succeed, you see the error message.

the other issue that can cause this is, not all bios's have a full sriov implementation that includes the ability to remap interrupts.

so the kvm parameter is used to allow this to work.