How to use VFs to get better performance with qat (is linear ) ?

[Henrylin1]

Hi Ronny,

Thanks for you reply.

I can't reply to you in https://community.intel.com/t5/Intel-QuickAssist-Technology/How-to-use-VFs-to-get-better-performance-with-qat-is-linear/m-p/1576598#M233 ,  maybe a bug? so I start a new post here to answer your questions.

Here are all my steps:

step1. build qat : ./configure --enable-icp-sriov=host; make install

step2.check service status : 

[root@localhost spdk]# service qat_service status
Checking status of all devices.
There is 34 QAT acceleration device(s) in the system:
 qat_dev0 - type: 4xxx,  inst_id: 0,  node_id: 0,  bsf: 0000:76:00.0,  #accel: 1 #engines: 9 state: up
 qat_dev1 - type: 4xxx,  inst_id: 1,  node_id: 1,  bsf: 0000:f3:00.0,  #accel: 1 #engines: 9 state: up
 qat_dev2 - type: 4xxxvf,  inst_id: 0,  node_id: 0,  bsf: 0000:76:00.1,  #accel: 1 #engines: 1 state: up
 qat_dev3 - type: 4xxxvf,  inst_id: 1,  node_id: 0,  bsf: 0000:76:00.2,  #accel: 1 #engines: 1 state: up
 qat_dev4 - type: 4xxxvf,  inst_id: 2,  node_id: 0,  bsf: 0000:76:00.3,  #accel: 1 #engines: 1 state: up
 qat_dev5 - type: 4xxxvf,  inst_id: 3,  node_id: 0,  bsf: 0000:76:00.4,  #accel: 1 #engines: 1 state: up
 qat_dev6 - type: 4xxxvf,  inst_id: 4,  node_id: 0,  bsf: 0000:76:00.5,  #accel: 1 #engines: 1 state: up
 qat_dev7 - type: 4xxxvf,  inst_id: 5,  node_id: 0,  bsf: 0000:76:00.6,  #accel: 1 #engines: 1 state: up
 qat_dev8 - type: 4xxxvf,  inst_id: 6,  node_id: 0,  bsf: 0000:76:00.7,  #accel: 1 #engines: 1 state: up
 qat_dev9 - type: 4xxxvf,  inst_id: 7,  node_id: 0,  bsf: 0000:76:01.0,  #accel: 1 #engines: 1 state: up
...

 step3.  PF and VF configuration files attached here , and I used up all 16vfs of 76:xx  

with scripts: 

for i in $(seq 1 7);do ./usertools/dpdk-devbind.py -b vfio-pci 0000:76:01.${i}; done
for i in $(seq 1 7);do ./usertools/dpdk-devbind.py -b vfio-pci 0000:76:00.${i}; done
./usertools/dpdk-devbind.py -b vfio-pci 0000:76:02.0 0000:76:01.0

and all  vf configuration files are same , vf \pf using 2 cores (0 and 1) for 2 dc instances

step4. run the command before starting spdk app

echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
rmmod usdm_drv
insmod /root/qat/quickassist/utilities/libusdm_drv/usdm_drv.ko max_huge_pages=1024 max_huge_pages_per_process=48

step5. start app

cd spdk
./build/examples/bdevperf -S 1 -q 64 -t 10 -C -m 0x3 -o 16384 -w randwrite -z -c ./test/compress/dpdk.json  &

with :  

[root@localhost spdk]# cat ./test/compress/dpdk.json
{
  "subsystems": [
    {
      "subsystem": "accel",
      "config": [
        {
          "method": "compressdev_scan_accel_module",
          "params": {
            "pmd": 1
          }
        }
      ]
    }
  ]
}

step6: create vbdev and run test

mount -t tmpfs -o size=10G tmpfs /mnt/tmpfs 
./scripts/rpc.py bdev_nvme_attach_controller -b nvme0n1 -t PCIe -a 0000:50:00.0
./scripts/rpc.py bdev_lvol_create_lvstore nvme0n1n1 lvs
./scripts/rpc.py bdev_lvol_create  -t -l lvs lv0 1024
./scripts/rpc.py bdev_compress_create -b lvs/lv0 -p /mnt/tmpfs
./examples/bdev/bdevperf/bdevperf.py perform_tests

[Ronny_G_Intel]

Hi Henrylin1,

Thank you very much for providing the requested information.

Let me look into it and I will get back to you as soon as possible.

By the way, I am going to add your original note to this community post and close the other one so that we avoid any possible duplication.

Regards,

Ronny G

[Ronny_G_Intel]

Original question posted here:  https://community.intel.com/t5/Intel-QuickAssist-Technology/How-to-use-VFs-to-get-better-performance-with-qat-is-linear/m-p/1576598#M233

Hello ,

When I do  data compression test with bdevperf ,  I can't get better IOPS improvement with 1 process and multi-threads or 2 vfs or more  . 

Platform infos :

Intel(R) Xeon(R) Platinum 8462Y+

lspci |grep Co-processor
76:00.0 Co-processor: Intel Corporation Device 4942 (rev 40)
78:00.0 Co-processor: Intel Corporation Device 2710
f3:00.0 Co-processor: Intel Corporation Device 4942 (rev 40)
f5:00.0 Co-processor: Intel Corporation Device 2710

Steps :

./build/examples/bdevperf -S 1 -q 64 -t 10 -C -m 0x3 -o 16384 -w randwrite -z -c ./test/compress/dpdk.json &

./scripts/rpc.py bdev_nvme_attach_controller -b nvme0n1 -t PCIe -a 0000:50:00.0
./scripts/rpc.py bdev_lvol_create_lvstore nvme0n1n1 lvs
./scripts/rpc.py bdev_lvol_create -t -l lvs lv0 1024
./scripts/rpc.py bdev_compress_create -b lvs/lv0 -p /mnt/tmpfs
./examples/bdev/bdevperf/bdevperf.py perform_tests

Results:

isal pmd :   IOPS 245621.32 for 1core 

qat pmd:   IOPS 190423.9 for 1 core  lower than soft accel 

[Ronny_G_Intel]

Hi Henrylin1,

I see that you are using Sapphire Rapids with driver version: QAT20.L.1.0.50-00003, checked the config files as well.

I would recommend that you access for telemetry data while the test is running, this will verify whether QAT is being used and how busy it is. 

Please refer to Monitoring Telemetry https://intel.github.io/quickassist/PG/infrastructure_device_telemetry.html#monitoring-telemetry-text-based

Please monitor device utilization while the test is being run and provide results.

Regards,

Ronny G

[Ronny_G_Intel]

Hi Henrylin1,

Just checking, did you have a chance to use Monitoring Telemetry https://intel.github.io/quickassist/PG/infrastructure_device_telemetry.html#monitoring-telemetry-text-based to monitor the device utilization while the test is running?

Regards,

Ronny G

[Ronny_G_Intel]

Hi Henrylin1,

I haven't heard back from you in a while so I will be closing the internal case number we created for this issue, please feel free to reopen if needed.

Regards,

Ronny G