Virtual Device creation for QAT endpoint is slow

[TusharAgarwal]

Hi, 

We are using QAT-1.7 and I am trying to create virtual devices for QAT endpoints using 

echo 16 > /sys/bus/pci/drivers/c6xx/0000:19:00.0/sriov_numvfs

above command, but it takes more than 10 seconds for these virtual devices to get created, why is it taking so long to do so?
Is it because that device driver resource is busy or something?
What are the options to improve this time?

[BrusC_Intel]

Hello, TusharAgarwal.

Thank you for posting on the Intel Community Support Forums.

I received your ticket regarding this particular QAT inquiry, and I will be assisting you with this.

Please allow me to review the version details and I will get back to you as soon as possible with a response or in case more information is required.

Regards,

Bruce C.

Intel Customer Support Technician 

[BrusC_Intel]

Hello, TusharAgarwal.

Thank you for waiting.

This particular behavior has not been observed nor reproduced. We can enable the virtual devices with the same command and it takes around 1 second (or probably less than that) to complete. Since this could not be reproduced, it doesn't seem related to the QAT driver, and there could be other platform-related details that may cause the command to take more time to complete.

For the test we used the latest QAT driver available in Download Center: 4.22.0-00001.

With no recommendations explicit recommendations to improve the times you are reporting, what we can do is to review how to setup QAT for virtualization, and we typically don't do this manually as in your case, but we run the following steps to install QAT for virtualization:

1. Run ./configure --enable-icp-sriov=host
2. Run make -j install
3. Run service qat_service_vfs start to start the virtual devices

Please let me know if this or a newer driver helps, or if we can help you with anything else, and I will follow up on June 9th in case you need additional time.

Regards,

Bruce C.

Intel Customer Support Technician

[TusharAgarwal]

Hi BrusC_Intel,

Let me provide you with the driver and QAT chipset which is being used in my setup.

The device which I'm using is running linux kernel v4.14, and has a QAT c62x chipset with 2 endpoints.

On each endpoint, I am trying to create 16 VFs, using QAT driver v1.7

Regards

Tushar Agarwal

[BrusC_Intel]

Hello, TusharAgarwal.

Thank you for the additional details.

Allow me to review and I will contact you back as soon as possible.

Regards,

Bruce C.

Intel Customer Support Technician

[BrusC_Intel]

Hello, TusharAgarwal.

Can you confirm if you followed the same steps we provided before to create the VFs?

1. Configure the QAT driver for virtualization by running the command: ./configure --enable-icp-sriov=host
2. Install the QAT driver by running the command: make -j install
3. Enable the VFs by running the command: service qat_service_vfs start 

We also ran the test with same QAT c62x chipset and same QAT driver for Hardware Version 1.7 (driver version 4.22.0-00001). Kernel was newer though: 5.17. This is why we suspect something else in your system or configuration could be causing this.

I will follow up on June 12th in case you require additional time.

Regards,

Bruce C.

Intel Customer Support Technician

[BrusC_Intel]

Hello, TusharAgarwal.

I wanted to follow up on this thread in case further assistance is still required.

I will follow up again on June 15th in case we do not hear from you.

Regards,

Bruce C.

Intel Customer Support Technician

[TusharAgarwal]

Hi Bruce C,

Before I follow the steps provided by you, I wanted to check few things with you.

Actually, I am building the QAT modules on some source machine and then exporting them to target machine where I need them, so what I wanted to ask is, if I follow the steps mentioned by you above, this will create the QAT driver modules in such a way that QAT VFs will already be present on the target machine every time I install these modules directly on that machine. And if I create the driver module itself in this manner, there is no need to create VFs everytime using the manual method which I suggested.

Please correct me if I am wrong anywhere.

Regards

Tushar Agarwal

[BrusC_Intel]

Hello, TusharAgarwal.

Thank you for the response.

Please allow us to confirm exactly how it will go is and I will get back to you as soon as possible.

Regards,

Bruce C.

Intel Customer Support Technician

[BrusC_Intel]

Hello, TusharAgarwal.

Thank you for waiting.

This exact scenario is not something we test because we always test the build of the QAT driver on the same machine where it's used

However, the QAT modules you want to build and export may be functional, although you would still have to enable the VFs at the end with the command service qat_service_vfs start in the target machine. That would be the last step from our previous instructions.

The first command (from our previous instructions) will configure the QAT driver for virtualization intended to be used from the Host. The second command will build the QAT modules using that configuration, so in this case, the modules will be built for a virtualization deployment in the Host. At this point, you may be able to export those modules to the target machine (again, this is not something we typically test, so we cannot validate it), and then run on the target machine the last command: service qat_service_vfs start

Something very important to keep in mind are the QAT configuration files that are created during installation. These configuration files are needed to properly use the QAT devices.

A good recommendation is to check the Getting Started Guide, section 3.2, as this includes the list of tasks that are done during the QAT driver installation.

If you have any other questions, please let me know, and I will follow up on June 16th in case you need additional time.

Regards,

Bruce C.

Intel Customer Support Technician

[TusharAgarwal]

Hi Bruce,

I was thinking of trying the above steps mentioned by you, I had one small doubt in the first step, I just checked the configure script and it has two options for enable_pci_sriov, one is host, and another is guest, which one should I use? Since as I mentioned earlier, I am creating these modules on a separate machine and then porting these modules to a target machine.

Regards

Tushar Agarwal

[BrusC_Intel]

Hello, TusharAgarwal.

Allow me to review if there is additional information we can provide on this regard, and I will get back to you

Regards,

Bruce C.

Intel Customer Support Technician

[BrusC_Intel]

Hello, TusharAgarwal.

Regarding your previous inquiry, when deploying QAT in virtualization environments, the deployment is done in two sides, the host and the guest OS(s). The QAT driver has to be installed in both sides. For the installation in the host system, the configuration flag used is --enable-icp-sriov=host. For the installation in the guest system, the configuration flag to use --enable-icp-sriov=guest. That's the difference about which one to use.

Your particular implementation, as commented before, is not really tested or validated by us, so the correct flag to use will depend on the role of the target system, will it be a host or a guest system? This will determine what you should should choose when installing the QAT driver.

If we can help with anything else, please let me know, and I will follow up on June 20th in case additional time is required.

Regards,

Bruce C.

Intel Customer Support Technician

[BrusC_Intel]

Hello, TusharAgarwal.

This post is just a quick follow up to check if we can help with anything else.

We will keep the thread open and will follow up again on June 23th in case you need additional time.

Regards,

Bruce C.

Intel Customer Support Technician

[BrusC_Intel]

Hello, TusharAgarwal.

This post is to inform you about the closure of this support thread.

We previously shared some details regarding the machine roles and how this particular implementation being attempted in your case is not validated/tested, but there was no confirmation about further assistance being required after trying to follow up.

The thread will be closed right now since there has been no response, and it will no longer be monitored by Intel support, but if you require any type of assistance from Intel in the future, just open a new thread.

Best regards,

Bruce C.

Intel Customer Support Technician