Ethernet Products
Intel® Ethernet products and technologies
3894 Discussions

XL710 -QDA2 SR-IOV problem:How to split different virtual function into different iommu_group

zengmou
Beginner
1,509 Views

Here is some information about my environment :

 

Hardware :

Intel(R) CPU Core i3 @ 3.90GHz;

Intel XL710-QDA2 Network Adapter;driver i40 and iavf is the latest version

 

System:

Centos 8

Kernel:4.18.0-240.22.1.el8_3.x86_64

Hypervisor:KVM

 

So far, vt-d and SR-IOV are all enabled on the BIOS and intel_iommu=on is enabled on the host system and it is verify by virt-host-validate command line.

 

XL710-QDA2 is virtualized into 1 virtual functions(vf). Things are getting different about iommu group.  however, vf from intel XL710-QDA2 are grouped into the same iommu.

it will the iommu_group as bellow:

<capability type='phys_function'>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>       //the PF  enp1s0f0
</capability>
<iommuGroup number='1'>
<address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>     //enp1s0f0
<address domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>     //the VF enp1s0f0vf0
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>     //enp1s0f1
</iommuGroup>

 

Since the vfs from intel XL710-QDA2 are in the same IOMMU group, whenever I want to attach enp1s0f0vf0 to VM, it will generate the error as below:

 

"

error: internal error: qemu unexpectedly closed the monitor: 2021-05-14T02:25:06.614553Z qemu-kvm: -device vfio-pci,host=0000:02:02.0,id=hostdev0,bus=pci.0,addr=0xc: vfio 0000:02:02.0: group 1 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

"

I have read some posts, saying that this is maybe the ACS problem but my cpu does support that:

"

Capabilities: [1d0 v1] Access Control Services

"

From Intel XL710-QDA2 specification, it says it supports SR-IOV, however, it does not work unless each VF has its own individual IOMMU group.Does anyone know how to do it?

 

Thank you!

Labels (1)
0 Kudos
17 Replies
Crisselle_Intel
Moderator
1,491 Views

Hello ezengmou,


Thank you for posting in Intel Ethernet Communities. 


Please share the following information for us to proceed checking your query.

1. Exact i40 and iavf driver version.

2. Firmware version of Intel XL710-QDA2 Network Adapter

3. Please share the PBA of the adapter. You may refer to the link below on where to find the PBA number. It consists 6-3 digit number located at the last part of the serial number. 

Identify Your Intel® Network Adapter Model Using PBA Number

https://www.intel.com/content/www/us/en/support/articles/000007022/network-and-i-o/ethernet-products...


Looking forward to your reply.


We will follow up after 3 business days in case we don't hear from you. 


Best regards,

Crisselle C.

Intel® Customer Support


zengmou
Beginner
1,466 Views

Hello Crisselle C

 

Thank you for your reply.

 

Here are some addtional information as bellow:

 

1、i40e version: 2.15.9

2、iavf version:version: 4.1.1

3、firmware-version: 6.01 0x800035da 1.1747.0

4、PBA numbe:J11363

 

Looking forward to your reply.

 

Best regards,

Zengmou.

 

 

 

 

Crisselle_Intel
Moderator
1,463 Views

Hello Zengmou,


You're welcome. Appreciate your effort in sending the requested information.


Please allow us to further check your query. We will give you an update as soon as possible but no later than 2-3 business days.


Hoping for your kind patience.


Best regards,

Crisselle C.

Intel® Customer Support


Crisselle_Intel
Moderator
1,447 Views

Hello Zengmou,

 

Good day!

 

We highly suggest checking this SR-IOV Configuration guide for XL710 on Linux and let us know if this would help or needs further assistance on this matter.

 

We look forward to your reply.

 

Should there be no response, we will follow up after 3 business days in case we don't hear from you. 

 

Best regards,

Crisselle C.

Intel® Customer Support

 

zengmou
Beginner
1,434 Views

Hello Crisselle C

 

Thank you for your reply.

 

I'm using the centos 8,and I have successfully created the VF, so i only tried this step:

 

18. The VF driver automatically loads in the host operating system as soon as the VFs are created by
the PF driver. The VF driver claims newly-created VFs, and these VF are not available for Virtual
Machine (VM) assignment. There are two methods to overcome this scenario:
a. Unload the VF driver from within host operating systems by executing the following command in
Linux terminal with superuser (root) permission.
# rmmod i40evf------------# is this driver  named as iavf in the latest version?
b. Blacklist VF driver by adding blacklist i40evf to the /lib/modprobe.d/dist-blacklist file, as shown in Figure 10.

 

I did these two step,but neither of them can split different virtual function into different iommu_group.

 

So do you have instructions for centos 8 ?

 

Looking forward to your reply.

 

Best regards,

Zengmou.

Crisselle_Intel
Moderator
1,411 Views

Hello Zengmou,


Thank you for trying out the recommendations.


To answer your question, i40evf - is this driver named as iavf in the latest version? Yes, it is now named as iavf. 


Pleas allow me to re-escalate this request to our higher level for further investigation. We will get back to you as soon as possible but no later than 2-3 business days. 


Best regards,

Crisselle C.

Intel® Customer Support


Crisselle_Intel
Moderator
1,400 Views

Hello Zengmou,


How are you doing?


This is just an update to inform you that we are still checking your request together with our engineers. Rest assured that we will give you an update as soon as we heard from them but no later than 2-3 business days.


Hoping for your kind patience.


Best regards,

Crisselle C.

Intel® Customer Support


zengmou
Beginner
1,391 Views

Hello Crisselle C

 

Thank you for your reply and hope to hear your good news。

 

Best regards,

Zengmou.

Crisselle_Intel
Moderator
1,359 Views

Hello Zengmou,


Thank you for your kind understanding.


We'd like to inform you that we are still actively working on this issue with our higher level Engineers. We are doing our best to sort this out as soon as possible. Rest assured that we will keep you posted and get back to you within 2-4 business days.


Best regards,

Crisselle C.

Intel® Customer Support


Crisselle_Intel
Moderator
1,332 Views

Hello Zengmou,


Good day!


Please be informed that this request is still escalated to our higher level engineers. We will give you an update as soon as we heard from them but no later than 2-4 business days.


Thank you for your kind patience.


Best regards,

Crisselle C.

Intel® Customer Support


zengmou
Beginner
1,325 Views

Hello Crisselle C

 

Thank you for your reply and hope to hear your good news。

 

Best regards,

Zengmou.

Crisselle_Intel
Moderator
1,307 Views

Hello Zengmou,


How are you doing? I hope everything is great!


We sincerely apologize for the delay on this matter as we are thoroughly checking on this together with our engineers. Please give us more time to look into this. Rest assured that we will give you an update as soon as there is any findings but not later than 3 business days.


Thank you for your kind understanding.


Best regards,

 

Crisselle Franz

Intel Customer Support


zengmou
Beginner
1,276 Views

Hello Crisselle C

 

Thank you for your reply and hope to hear your good news。

 

Best regards,

Zengmou.

Crisselle_Intel
Moderator
1,261 Views

Hello Zengmou,


Apologies for the delay on this request. 


Please be informed that we are currently seeking assistance from our virtualization team. Rest assured that we will give you an update as soon as we heard from them but no later than 2-4 business days.


Thank you for your kind understanding.


Best regards,

Crisselle C.

Intel® Customer Support 


Crisselle_Intel
Moderator
1,238 Views

Hello Zengmou,


Thank you for the patience on this matter.


Please see below update from our engineering level. Feel free to let us know if you have additional questions and clarifications on this matter. 


We believe the issue is due to the CPU. From everything they have been able to find, Core i3's do not have ACS support.

1) Could you please send the full output of lspci -vv so that they can check this?

You can use the command lspci -vv > lspci.txt to put the output into a file.

2) You may also try adding iommu=pt to the kernel options (same place where intel_iommu=on should be added).

They believe this won't fix the issue, but it doesn't hurt to try.


Looking forward to your reply.


We will follow up after 3 business days in case we don't hear from you.


Best regards,


Crisselle Franz

Intel Customer Support


Crisselle_Intel
Moderator
1,224 Views

Hello Zengmou,


Good day!


This is just a follow up to check if you received our previous update. If yes, kindly provide the additional information requested for us to continue checking on this query. In case you have additional questions or clarifications, please do not hesitate to ask.


Hoping to hear from you soon.


Should there be no response from you, we will follow up after 3 business days.

 

Best regards,

Crisselle Franz

Intel Customer Support


Crisselle_Intel
Moderator
1,186 Views

Hello Zengmou,


How are you doing?


Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future as this thread will no longer be monitored.


Thank you for choosing Intel and stay safe!


Best regards,

Crisselle Franz

Intel Customer Support


Reply