Intel XHCI controller with SR-IOV support (XHCI-IOV)

[FKupe]

Hello,

I'd like to use SR-IOV for the virtualization of an XHCI host controller to passthrough to multiple VMs. Software USB passthrough is not an option in this setup, which is why I'm looking into SR-IOV.

According to the XHCI specification document (I accessed it via https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interfac… ) in Section 8 "Virtualization", there should be a possibility to use SR-IOV for XHCI controllers.

I tried on a system with Intel C610 server chipset, with SR-IOV enabled (tested the SR-IOV support and it is working with an I710 network card), to use it with the XHCI controller on the Chipset, but that one doesn't show the PCI express capability for SR-IOV.

I'm wondering if there is a chipset or external XHCI controller on which SR-IOV is actually usable? Or does Linux just not show the capability although the hardware is capable? Is there support for it in the Linux drivers (it doesn't seem so at a first glance on the code)? What is the plan on proceeding with XHCI IO Virtualization in the future?

Thanks a lot for your help.

Best Regards

Felix

[idata]

Hello FelixK,

I was reviewing the information that you are looking for and I would like to provide you the most accurate information in order to assist you with this. Moreover, before to provide you more information I would like to make sure some details of your system. Please provide us the following information:

A) Motherboard.

B) Processor.

C) Operating System.

Also, I would like to know why the USB Passthrough is not an option? It is due to the operating system does not have the compatibility or because you will need to use specifically the XHCI-IOV?

Have you ever tried to setup this feature with Red-Hat or Proxmox Virtual Environment?

I will share with you some link that I found related to SR-IOV and XHCI on Red Had and Proxmox that could help you with this configuration. *Those links are not from Intel*

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-pci_devices-pci_passthrough

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/virtualization_deployment_and_administration_guide/sect-kvm_guest_virtual_machine_compatibility-usb3.0-support

https://pve.proxmox.com/wiki/USB_physical_port_mapping

I will be waiting the outcome in order to proceed with the next step. If you have any other questions please do not hesitate and reply and I will be more than happy to assist you.

Best Regards,

Emeth X

[FKupe]

Hello Emeth,

thanks for your reply!

The hardware used in this setup is:

• Supermicro X10SRL-F Mainboard
• Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz
• Cent OS 7 with a 4.4 Linux kernel (could be changed to any kernel, I don't mind if the result is working...)

I tested the SR-IOV capability of this setup using the X710-2 network card. SR-IOV worked exactly as it should, so the BIOS/Mainbaord/Chipset seem to support it properly. It was just impossible to use it with the XHCI USB controller on the Chipset.

USB passthrough in the traditional software way is not an option because of several reasons:

• First, the USB 3 performance is needed and some software solutions seem to be slow down the connection in some cases or even only support USB 2 emulation (but qemu does support virtual USB3)
• I need to passthrough a whole USB port or USB hub port with any (or even no) connected device, so that the plugging events will be handled by the guest OS. The plan is to just pass through the port and then a device can be (un)plugged at any time, all of it being handled by the guest kernel only.
• I tried using the corresponding qemu functionality, but it created hot-plug events on the host first, and only then hooked the device into the guest. This makes the device think it was re-plugged, so by just plugging in the device physically once, the device gets plug + unplug + plug.
• I actually also have a driver issue with some guests, which would provide drivers for Intel XHCI but not for NEC XHCI/emulated Qemu ones... However VirtualBox seems to offer Intel XHCI emulation so I could go for this one. But VirtualBox seems to also have the plug (Linux driver inits the device) + unplug + plug (guest driver comes second) problem, but I'll probably also try it with Virtualbox more which I didn't do yet.
• Generally the XHCI SR-IOV support seemed like the perfect solution for this, I just couldn't practically set it up.

So I tried the virtual USB passthrough using Qemu + libvirt and also plain Qemu (with the same approach on the qemu command line which also your link https://pve.proxmox.com/wiki/USB_physical_port_mapping https://pve.proxmox.com/wiki/USB_physical_port_mapping suggests; using: -device usb-host,hostbus=x,hostport=x.1). Only the latter supported passing through a whole port, libvirt only supports an individual device. I also tried udev hooks and several workarounds but never got rid of the device attaching to the host OS first and getting setup by the Linux kernel...

So the goal is to have the device invisible to the host and only the guest kernel communicates with it, while still being able to do this with multiple guests on different physical USB ports (e.g. 6 Ports - 6 Guests). I also consider external PCI devices to ramp up the port number, but it seems like a bad solution to buy a whole PCIe XHCI controller for each guest. This would be one of the possible fallbacks though.

I'd also like to know if there is an external XHCI chip from Intel which can be attached as a PCIe device to a server. Most cards I found seem to be using NEC chips there...

Thanks a lot for your help!

Best Regards

Felix

[idata]

Hello FelixK,

Thank you so much for the information provided.

Due to the fact that you are using a SuperMicro Server Board my best recommendation in this scenario is to contact the system manufacture in order to make sure those details. Because in some OEM boards the manufacture disable some features of the chipset.

Best Regards,

Emeth X

[FKupe]

Hello Emeth,

thanks for your reply. While it could be an issue of the mainboard/firmware provided by SuperMicro, I'd still be interested to know more about the following things:

• is there an external (not on chipset) XHCI controller from Intel? As that worked very well with the network card, it should work with other external PCIe devices, too.
• How does the virtualization of the Intel chipset's XHCI controller with SR-IOV work on Linux? I couldn't find much documentation except for the reference in the specification...
• Is there an Intel mainboard or reference system where I could try to use SR-IOV with XHCI? Any hardware you could suggest for this purpose?

I'll also contact SuperMicro in case their firmware hides a capability of the chipset.

Best Regards

Felix

[idata]

Hello FelixK,

I would like to recommend you to explained them the information that you are looking on this matter, because as I mention before if you were able to set it up using the Network Adapter XL710 there are some features in your system that are enable o disable from the Intel Chipset.

Best Regards,

Emeth X

[FKupe]

Hello Emeth X,

I am already in contact with the SuperMicro support in this case. My three questions above were no longer related to this specific mainboard, as I consider buying any Intel hardware which will support this use case.

The specific mainboard was just "lying around" an therefore used in my test, but as it is only an evaluation phase I can try as much hardware/software toolstack as needed to get the use case sorted.

Can you please give me some advice on these three questions, esp. also the third question, which mainboard from Intel officially supports this use case?

Thank you and best Regards

Felix

[FKupe]

Hello,

I now got some detailed information from the SuperMicro BIOS developers, which were very helpful in this case.

According to them, the C612 EDS does not support the two features Access Control Services (ACS) and Alternate Routing ID (ARI) on the PCIe root port of the on-chipset devices.

This is why SR-IOV is not supported for the on-chipset devices and specifically the XHCI controller on the C612 chipset.

I'm still looking for hardware which actually supports SR-IOV with an XHCI controller (internal or external), preferably an Intel XHCI controller for broader guest driver support.

Do you maybe have an internal test system for this? Is there anyone else who did setup one XHCI controller on multiple VMs using SR-IOV?

Thanks!

Felix

[idata]

Hello,

Thank you so much for the information provided,

Let me investigate more about the Intel Hardware that will support those features.

As soon as I have more information of this I will let you know.

Best Regards,

Emeth X

[idata]

Hello,

Thank you so much for the time that you have been waiting about this information.

I would like to share with you some link where you will find more information about this feature, please check the links below:

https://www.intel.com/content/www/us/en/support/articles/000005722/network-and-i-o/ethernet-products.html?_ga=2.10219189.1407945610.1535426500-828520152.1535068798&elq_cid=4335954 Frequently Asked Questions for SR-IOV on Intel® Ethernet Server Adaptershttps://www.intel.com/content/www/us/en/support/articles/000007947/server-products.html Single Root I/O Virtualization Support for Intel® Server Board S2600GZ/GL Familieshttps://www.intel.com/content/www/us/en/pci-express/pci-sig-single-root-io-virtualization-support-in-virtualization-technology-for-connectivity-paper.html Enhance I/O Performance with Intel® Virtualization TechnologyI hope the information will be informative for you.

Best Regards,

Emeth O

[idata]

Hello,

I would like to know if you have any other question in order to assist you with. If so, please do not hesitate and let us know and we will be more than happy to assist you.

Best regards,

Emeth O

[FKupe]

Hello Emeth,

thank you for following up and providing some information. The list of mainboards mentions that SR-IOV is supported with the onboard LAN adapter, which at least means it supports SR-IOV for certain onboard devices.

Can you confirm if the onboard USB3 controller (XHCI) also supports SR-IOV and provide me some references on how it is used / tested?

Best Regards

Felix

[idata]

Hello,

At this point, the best recommendation would be to contact the manufacture OEM (SuperMicro) this because Intel provides the chipset by each OEM. So, they can change the original configuration so it would fit their systems.

Best regards,Emeth O

[FKupe]

Hello Emeth,

thanks for your support in this thread and the help you have provided so far. I will try to get support from manufacturers on possible alternative hardwares.

Best Regards

Felix