XL710 -QDA2 SR-IOV problem:How to split different virtual function into different iommu_group

[zengmou]

Here is some information about my environment :

Hardware :

Intel(R) CPU Core i3 @ 3.90GHz;

Intel XL710-QDA2 Network Adapter;i40 and iavf is the latest version

System:

Centos 8

Kernel:4.18.0-240.22.1.el8_3.x86_64

Hypervisor:KVM

So far, vt-d and SR-IOV are all enabled on the BIOS and intel_iommu=on is enabled on the host system and it is verify by virt-host-validate command line.

XL710-QDA2 is virtualized into 1 virtual functions(vf). Things are getting different about iommu group.  however, vf from intel XL710-QDA2 are grouped into the same iommu.

it will the iommu_group as bellow:

<capability type='phys_function'>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>       //the PF  enp1s0f0
</capability>
<iommuGroup number='1'>
<address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>     //enp1s0f0
<address domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>     //the VF enp1s0f0vf0
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>     //enp1s0f1
</iommuGroup>

Since the vfs from intel XL710-QDA2 are in the same IOMMU group, whenever I want to attach enp1s0f0vf0 to VM, it will generate the error as below:

"

…

error: internal error: qemu unexpectedly closed the monitor: 2021-05-14T02:25:06.614553Z qemu-kvm: -device vfio-pci,host=0000:02:02.0,id=hostdev0,bus=pci.0,addr=0xc: vfio 0000:02:02.0: group 1 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

…

"

I have read some posts, saying that this is maybe the ACS problem but my cpu does support that:

"

…

Capabilities: [1d0 v1] Access Control Services

…

"

From Intel XL710-QDA2 specification, it says it supports SR-IOV, however, it does not work unless each VF has its own individual IOMMU group.Does anyone know how to do it?

Thank you!