Intel® QuickAssist Technology (Intel® QAT)
For questions and discussions related to Intel® QuickAssist Technology (Intel® QAT).
31 Discussions

How to use VFs to get better performance with qat (is linear ) ?

Henrylin1
Beginner
560 Views

Hi Ronny,

Thanks for you reply.

I can't reply to you in https://community.intel.com/t5/Intel-QuickAssist-Technology/How-to-use-VFs-to-get-better-performance-with-qat-is-linear/m-p/1576598#M233 ,  maybe a bug? so I start a new post here to answer your questions.

 

Here are all my steps:

step1. build qat : ./configure --enable-icp-sriov=host; make install

step2.check service status : 

 

[root@localhost spdk]# service qat_service status
Checking status of all devices.
There is 34 QAT acceleration device(s) in the system:
 qat_dev0 - type: 4xxx,  inst_id: 0,  node_id: 0,  bsf: 0000:76:00.0,  #accel: 1 #engines: 9 state: up
 qat_dev1 - type: 4xxx,  inst_id: 1,  node_id: 1,  bsf: 0000:f3:00.0,  #accel: 1 #engines: 9 state: up
 qat_dev2 - type: 4xxxvf,  inst_id: 0,  node_id: 0,  bsf: 0000:76:00.1,  #accel: 1 #engines: 1 state: up
 qat_dev3 - type: 4xxxvf,  inst_id: 1,  node_id: 0,  bsf: 0000:76:00.2,  #accel: 1 #engines: 1 state: up
 qat_dev4 - type: 4xxxvf,  inst_id: 2,  node_id: 0,  bsf: 0000:76:00.3,  #accel: 1 #engines: 1 state: up
 qat_dev5 - type: 4xxxvf,  inst_id: 3,  node_id: 0,  bsf: 0000:76:00.4,  #accel: 1 #engines: 1 state: up
 qat_dev6 - type: 4xxxvf,  inst_id: 4,  node_id: 0,  bsf: 0000:76:00.5,  #accel: 1 #engines: 1 state: up
 qat_dev7 - type: 4xxxvf,  inst_id: 5,  node_id: 0,  bsf: 0000:76:00.6,  #accel: 1 #engines: 1 state: up
 qat_dev8 - type: 4xxxvf,  inst_id: 6,  node_id: 0,  bsf: 0000:76:00.7,  #accel: 1 #engines: 1 state: up
 qat_dev9 - type: 4xxxvf,  inst_id: 7,  node_id: 0,  bsf: 0000:76:01.0,  #accel: 1 #engines: 1 state: up
...

 

 step3.  PF and VF configuration files attached here , and I used up all 16vfs of 76:xx  

with scripts: 

for i in $(seq 1 7);do ./usertools/dpdk-devbind.py -b vfio-pci 0000:76:01.${i}; done
for i in $(seq 1 7);do ./usertools/dpdk-devbind.py -b vfio-pci 0000:76:00.${i}; done
./usertools/dpdk-devbind.py -b vfio-pci 0000:76:02.0 0000:76:01.0

and all  vf configuration files are same , vf \pf using 2 cores (0 and 1) for 2 dc instances

step4. run the command before starting spdk app

echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
rmmod usdm_drv
insmod /root/qat/quickassist/utilities/libusdm_drv/usdm_drv.ko max_huge_pages=1024 max_huge_pages_per_process=48

step5. start app

cd spdk
./build/examples/bdevperf -S 1 -q 64 -t 10 -C -m 0x3 -o 16384 -w randwrite -z -c ./test/compress/dpdk.json  &

with :  

[root@localhost spdk]# cat ./test/compress/dpdk.json
{
  "subsystems": [
    {
      "subsystem": "accel",
      "config": [
        {
          "method": "compressdev_scan_accel_module",
          "params": {
            "pmd": 1
          }
        }
      ]
    }
  ]
}

step6: create vbdev and run test

mount -t tmpfs -o size=10G tmpfs /mnt/tmpfs 
./scripts/rpc.py bdev_nvme_attach_controller -b nvme0n1 -t PCIe -a 0000:50:00.0
./scripts/rpc.py bdev_lvol_create_lvstore nvme0n1n1 lvs
./scripts/rpc.py bdev_lvol_create  -t -l lvs lv0 1024
./scripts/rpc.py bdev_compress_create -b lvs/lv0 -p /mnt/tmpfs
./examples/bdev/bdevperf/bdevperf.py perform_tests

 

0 Kudos
5 Replies
Ronny_G_Intel
Moderator
504 Views

Hi Henrylin1,


Thank you very much for providing the requested information.

Let me look into it and I will get back to you as soon as possible.


By the way, I am going to add your original note to this community post and close the other one so that we avoid any possible duplication.


Regards,

Ronny G


0 Kudos
Ronny_G_Intel
Moderator
500 Views

Original question posted here:  https://community.intel.com/t5/Intel-QuickAssist-Technology/How-to-use-VFs-to-get-better-performance-with-qat-is-linear/m-p/1576598#M233

 

Hello ,

 

When I do  data compression test with bdevperf ,  I can't get better IOPS improvement with 1 process and multi-threads or 2 vfs or more  . 

 

Platform infos :

Intel(R) Xeon(R) Platinum 8462Y+

lspci |grep Co-processor
76:00.0 Co-processor: Intel Corporation Device 4942 (rev 40)
78:00.0 Co-processor: Intel Corporation Device 2710
f3:00.0 Co-processor: Intel Corporation Device 4942 (rev 40)
f5:00.0 Co-processor: Intel Corporation Device 2710

 

Steps :

./build/examples/bdevperf -S 1 -q 64 -t 10 -C -m 0x3 -o 16384 -w randwrite -z -c ./test/compress/dpdk.json &

./scripts/rpc.py bdev_nvme_attach_controller -b nvme0n1 -t PCIe -a 0000:50:00.0
./scripts/rpc.py bdev_lvol_create_lvstore nvme0n1n1 lvs
./scripts/rpc.py bdev_lvol_create -t -l lvs lv0 1024
./scripts/rpc.py bdev_compress_create -b lvs/lv0 -p /mnt/tmpfs
./examples/bdev/bdevperf/bdevperf.py perform_tests

 

Results:

isal pmd :   IOPS 245621.32 for 1core 

qat pmd:   IOPS 190423.9 for 1 core  lower than soft accel 

0 Kudos
Ronny_G_Intel
Moderator
497 Views

Hi Henrylin1,


I see that you are using Sapphire Rapids with driver version: QAT20.L.1.0.50-00003, checked the config files as well.

I would recommend that you access for telemetry data while the test is running, this will verify whether QAT is being used and how busy it is. 

Please refer to Monitoring Telemetry https://intel.github.io/quickassist/PG/infrastructure_device_telemetry.html#monitoring-telemetry-text-based

Please monitor device utilization while the test is being run and provide results.


Regards,

Ronny G


0 Kudos
Ronny_G_Intel
Moderator
343 Views

Hi Henrylin1,


Just checking, did you have a chance to use Monitoring Telemetry https://intel.github.io/quickassist/PG/infrastructure_device_telemetry.html#monitoring-telemetry-text-based to monitor the device utilization while the test is running?


Regards,

Ronny G


0 Kudos
Ronny_G_Intel
Moderator
185 Views

Hi Henrylin1,


I haven't heard back from you in a while so I will be closing the internal case number we created for this issue, please feel free to reopen if needed.


Regards,

Ronny G


0 Kudos
Reply