Build a server with virtual GPU support (no passthrough)

[emaroudas]

Hello,

I would like to build a Linux-based server machine able to provide GPU virtualization support for QEMU/KVM-based virtual machines (VMs), in order for each VM to offload computations to its assigned virtual GPU (on top of the integrated graphics of the processor, no passthrough the whole device). Obviously, the more VMs that can run simultaneously, the better.

My understanding so far is that both the CPU and motherboard should support virtualization technologies such as VT-d and SR-IOV.

For CPU selection, there are both server- and desktop-oriented processors that support the necessary virtualization technologies. As for now, I am considering Intel Xeon processors along these specs [ark.intel].

The motherboard selection seems to be a bit more difficult to me. According to various online discussions (i.e here, here and here), the aperture size option in the BIOS/UEFI is a critical factor that implicitly controls the number of distinct virtual GPUs (mdev devices in linux) available to the hypervisor for assignment to the VMs.

The above links mention a few motherboards that allow the user to increase the aperture size reserved from the BIOS for the GPU, but I cannot find a reliable way to search among motherboards for that specific feature, apart from searching on each manual individually.

Also, many of them seem to either hide that option or it is not working as expected, i.e. the number of virtual GPUs does not increase when the aperture size increases.

The above links also discuss some "dirty" methods (i.e. modify BIOS, grub) to increase the aperture size to a maximum of 2048 MB, but I would prefer a "non-hacky" solution if possible.

Overall, my questions are:

1) Is there a CPU / motherboard combination that it is known to provide the maximum number of virtual GPUs? Any pointers on how to identify such a pair?

2) Are there any "prebuilt" workstations that are suitable for the use-case I am interested in?

3) Are there any other configuration options that affect the number of available virtual GPUs, like linux driver options, or other hardware/software knobs that I am missing?

Any advice, comment or suggestion on the hardware selection or narrowing it down, would help.

PS.

I am looking for a solution apart from NVIDIA cards, due to the overall cost and no access to the source code of the driver/runtime.

Thanks,

Emmanouil Maroudas

[Victor_G_Intel]

Hello emaroudas,

Thank you for posting on the Intel® communities.

Please find your questions answered down below:

1-Is there a CPU / motherboard combination that is known to provide the maximum number of virtual GPUs? Any pointers on how to identify such a pair?

R/ When it comes to a motherboard /CPU combination, please take into consideration that we don’t focus on virtualization capabilities when we are testing our components. What we do, is focused on basic and functional compatibility, so components can potentially be chosen by a person or a group of persons in order to build up a server system.

2-Are there any "prebuilt" workstations that are suitable for the use case I am interested in?

R/If we talk about workstations only, we actually don’t have any particular ones tested on our end, especially since we don’t really make them ourselves; therefore, if you find one that suits your needs or just caught your attention from any third party’s brand, our best recommendation will be to clear out any questions with that third party brand directly.

What we do offer in terms of systems are our Intel® Data Center Blocks (Intel® DCB); however, these are more focused on enterprise-related solutions; nonetheless, in case that you are interested in a solution for a company, this might be a great way to simplify things. For more information on our Intel® Data Center Blocks (Intel® DCB), you can check the links down below.

Intel® Data Center Blocks (Intel® DCB)

https://www.intel.com/content/www/us/en/products/details/servers/data-center-blocks.html

Intel® Data Center Blocks for Cloud (Intel® DCB for Cloud)

https://www.intel.com/content/www/us/en/products/details/servers/data-center-blocks/dcb-cloud.html

Intel® Data Center Blocks (Intel® DCB) for Nutanix Enterprise Cloud Platform

https://www.intel.la/content/www/xl/es/products/docs/servers/data-center-blocks/dcb-cloud/nutanix-enterprise-cloud.html

3-Are there any other configuration options that affect the number of available virtual GPUs, like Linux driver options, or other hardware/software knobs that I am missing?

R/ For more information about Virtual CPU’s please take a look at the information in the link down below, and in case of any doubt please remember that you can always contact the developer of the OS distribution that you are interested in using for virtualization.

How Many Virtual Processors (vCPUs) are Within an Intel® Xeon® Processor?

https://www.intel.com/content/www/us/en/support/articles/000036898/processors/intel-xeon-processors.html

Regards,

Victor G.

Intel Technical Support Technician

[emaroudas]

Hi Victor,

Thank you for the provided info.

Intel DCB seems too much for the intended use-case I have in mind, a standalone server machine mostly for Lab usage, where each VM would have access to its own virtual GPU, in order to offload computations in an isolated environment.
From my search so far, my understanding is that all latest Intel processors (11th generation) and chipsets (B-, Q-, H-, Z-series) support virtualization for both the CPU and the integrated GPU (VT-x, VT-d). The critical part is that the combination of them (CPU + chipset) and configuration (in BIOS/UEFI) seems to affect the virtualization capacity of the integrated GPU (for which I am mostly interested).

In particular,

for the motherboard:
AFAICT, the Z-series of chipsets, due to the overclocking support, offers the most configuration options in the BIOS/UEFI menu, so I guess it would also allow the increase of the aperture size. Is this full-access-to-all-BIOS-options mandatory for every motherboard with a Z-series chipset, or it is a motherboard-vendor's decision to offer it or not? If not, are there any vendors known to offer an unlocked BIOS version?

For the CPU:
A closer comparison between Intel Xeon and Intel Core processors (11th generation) in terms of virtualization and integrated graphics features seems quite similar. Also the fact that I cannot find any additional information (regarding my use-case of virtual GPUs) about Xeon processors or server chipsets, shifts my decision more to the Core family.

As of this time, I am thinking of the following hardware:
Motherboard: something with the Z490 chipset (possibly ASUS, MSI, Gigabyte or other)
CPU: Intel® Core™ i9-11900K Processor

Of course, this is not a final decision, just a combination of what IMHO should work (not 100% sure either).
If anyone with a similar or different setup can provide more info, tht would be great.

Forgive me if I sound too confused, I am still trying to get the whole picture of all these dependencies/interactions and configuration among the various components involved (CPU, motherboard, BIOS) to avoid ending up with an unsuitable setup.

Best,
Emmanouil Maroudas

[Victor_G_Intel]

Hello emaroudas,

Thank you for your response.

In regard to the following question: Is this full-access-to-all-BIOS-options mandatory for every motherboard with a Z-series chipset, or it is a motherboard vendor's decision to offer it or not? If not, are there any vendors known to offer an unlocked BIOS version?

R/Please take into consideration, that we are not familiar with how other manufacturers design, develop and integrate their BIOS; therefore, if you already have a motherboard or set of motherboards in mind, feel free to contact the specific manufacturer or manufacturers in order to get more information about the options included in their BIOS when it comes to virtualization.

Regards,

Victor G.

Intel Technical Support Technician  

[emaroudas]

Hi Victor,

Thanks for the clarification and the fast response.

I will contact directly with the motherboard manufacturers to gather more info about specific products.

Best,

Emmanouil Maroudas

[Victor_G_Intel]

Hello emaroudas,

Thank you for your response.

We wish you the best of lucks with either one of the manufacturers that you get in touch with, and with the whole project that you are building. If you don’t have any more questions, feel free to leave the thread open, in case that any community member decides to jump in to provide any advice for you. In case that you need anything else please let us know.

Regards,

Victor G.

Intel Technical Support Technician  

[Victor_G_Intel]

Hello emaroudas,

I hope that you are doing okay.

Since we referred you to contact the manufacturer of the motherboard or motherboards that you were interested in, this thread will no longer be monitored; however,  if you have any Intel related questions in the future we will gladly assist you, you can always submit a new question, and we will proceed with the assistance accordingly.

Best regards,

Victor G. 

Intel Technical Support Technician