82576 Gigabit Network adapter SR-IOV problem:How to split different virtual function into different iommu group?

[ali13]

Here is some information about my environment :

Hardware :

Supermicro X10SRi-F motherboard with Onboard i350 controller;

Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz;

Intel Corporation 82576 Gigabit Network Adapter;

System:

Centos 7

Kernel:4.15.4-1.el7.elrepo.x86_64

Hypervisor:KVM

So far, vt-d and SR-IOV are all enabled on the BIOS and intel_iommu=on is enabled on the host system and it is verify by virt-host-validate command line.

Both onboard i350 controller and intel 82576 are virtualized into different virtual functions(vf). Things are getting different about iommu group. Each vf from i350 could successfully be in individual iommu group, but vf from intel 82576 are grouped into the same iommu.

Since all the vfs from intel 82576 are in the same IOMMU group, whenever I want to attach any of them to VM, it will generate the error as below:

"

…

2018-02-23T17:16:18.202445Z qemu-kvm: -device vfio-pci,host=04:10.1,id=hostdev0,bus=pci.0,addr=0x9: vfio: error, group 34 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

…

"

As for the i350, I don't have this problem.

I have read some posts, saying that this is maybe the ACS problem but my cpu does support that:

"

00:01.0 PCI bridge: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1 (rev 01) (prog-if 00 [Normal decode])

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+

….

Capabilities: [100 v1] Vendor Specific Information: ID=0002 Rev=0 Len=00c <?>

Capabilities: [110 v1] Access Control Services

…

"

From Intel 82576 specification, it says it supports SR-IOV, however, it does not work unless each VF has its own individual IOMMU group.Does anyone know how to do it?

Thank you!

[idata]

Hi Happyava,

Thank you for posting in Wired Communities. Can you double check if the 82576 Gigabit network adapter you refer here is an onboard NIC embedded on the system or a standalone network adapter? What is the exact model of this network adapter? You may refer to the list below to check:

https://ark.intel.com/products/family/46829/Intel-Gigabit-Server-Adapters https://ark.intel.com/products/family/46829/Intel-Gigabit-Server-Adapters

Thanks,

Sharon T

[ali13]

Hi,Sharon:

82576 Gigabit network adapter I refer here is a standalone network adapter. I plug it into PCI-e slot on the MB X10SRI-F from supermicro.

I am still working on finding out the exact model of this network adapter.

If i use lspci, it shows "Intel Corporation 82576 Gigabit Network Connection"

And i also found the Serial Number is "1CA2CE0341ADG18771-002" on the network adapter.

Thanks,

[idata]

Hi Happyava,

Thank you for the information, you had the same inquiry in other post, I will just answer you from here. Based on the serial number G18771-002 this is an ET Quad port Server adapter but is an OEM unit. The part number G18772 is not on the retail unit list at https://www.intel.com/content/www/us/en/support/articles/000005612/network-and-i-o/ethernet-products.html. Do you happen to know from which OEM vendor this card was purchase? By searching online, this part seems to be belong to Dell. Thanks,Sharon T

[ali13]

Hi, Sharon:

Yes, this is a unit from our Dell server.

Can I ask, if we have a network adapter with a serial number, how do we find out the version or device number from Intel website by ourselves?

Thank you

[idata]

Hi Happyava,

Thank you for the confirmation. You may refer to the website below on how you can find the device ID based on the different operating system. https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products.html If it is an Intel retail network adapter, you can first identify the network adapter model by checking the serial number on the white sticker of the physical NIC. Here is the sample serial number https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products.html Look for the last 6-3 digits, C80235-001 is the sample here, then go to this website "List of Supported Retail Intel Ethernet Adapters" https://www.intel.com/content/www/us/en/support/articles/000005612/network-and-i-o/ethernet-products.html Search "C80235" on this website, you will find the corresponding Part number, vendor and device ID. I hope I was able to clarify your concern. Have you also tried contacting Dell to check with them about this OEM NIC?Regards,Sharon T

[ali13]

Hi, Sharon:

Thank you so much for your support. They are very helpful!

I have not got any responses from Dell yet but today when we replaced the 82576 NIC with i350 network adapter, it works now.

Again, thanks a lot for your support.

Regards,

Ava

[idata]

Hi Happyava,

Thank you for the update and glad to be of help. Glad to know I350 netwokr adapter works for you. Regards,Sharon T