devices in same iommu_group error with SRIOV NIC X540-T2 when KVM VM starts

[懷曾]

Hi

I am trying SRIOV NIC on my computer(intel X540-T2)

everything is fine, I can see VF and PF in PCI list but when I assigned 1 VF to 1 VM and then boot it, I got an error message

here is the message

Internal error: process exited while connecting to monitor: 2017-04-11T04:32:00.179452Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. 2017-04-11T04:32:00.179495Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 1 2017-04-11T04:32:00.179502Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed. 2017-04-11T04:32:00.179511Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 88, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 124, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1404, in startup self._backend.create() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1035, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: process exited while connecting to monitor: 2017-04-11T04:32:00.179452Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver. 2017-04-11T04:32:00.179495Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 1 2017-04-11T04:32:00.179502Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed. 2017-04-11T04:32:00.179511Z qemu-kvm: -device vfio-pci,host=02:10.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized

I think the reason is all of the VF and PF devices are in the same iommu_group and I found they are really in the same iommu_group after I check that.

It seems that all devices in same iommu_group should be assigned to 1 VM, if not, it will cause an error.

However I need to assign each VF to different VM.

Now I temporary move the NIC to a server and the problem is gone but because of some reason I prefer to use this NIC on my original computer.

Does this have solution? thanks a lot.

My environment :

Host OS : RHEL 7.3 Kernel Version 3.10.0-514.10.2.el7.x86_64

GNOME 3.14.2

CPU: I7-6700

MB:ASUS Q170

PF driver : ixgbe 5.0.4

kernel command line : intel_iommu=on pci=realloc pci=assign-busses

VM OS : fedora 25

VF driver : ixgbevf 4.0.3

[idata · Accepted solution]

Hi w180112,

Thank you for the post. Please check if your board supports VT-D and SR-IOV. It is recommended to use server board that supports VT-D and SR-IOV. Thanks,wb

[懷曾]

Hi wb

thanks for the reply

I have set VT-d and VT-x enable in bios.

So it's no solution unless I change the environment to a server right?

thanks

[idata]

Hi w180112,

Thank you for the reply. Let me further check.rgds,wb

[idata]

Hi w180112,

Yes. You need to use server board that supports VT-D and SR-IOV. Hope this clarified.Rgds,wb

[idata]

Hi W180112,

Please feel free to update if further clarification needed. Thanks. Rgds,wb

[懷曾]

sorry for late reply

I am sure that VT-D and VT-x is enable and iommu is on

however I still got error :(

maybe i should give up for using this card on my original PC?

thanks

[idata]

HI W180112,

You need to use a server board that supports VT-D and SR-IOV. Thank you. Rgds,wb