How to perform a memmove operation using DSA(data streaming accelerator) in linux kernel?

[yun3319]

Hello,

I am currently working on a kernel module where I need to perform a memmove operation using Data Streaming Accelerator (DSA).

I am using a 4th generation or later Intel Xeon processor, and I have successfully verified that DSA works correctly in user space by following the examples provided in the user guide. While I have been able to find examples and documentation for using DSA in user space, I am struggling to implement the same operation within the kernel.

Could someone please provide a clear, step-by-step guide or example code on how to correctly set up and execute a memmove operation using DSA from within the kernel?

Any help or guidance would be greatly appreciated. Thank you!

[RNa]

You can try this using the Linux dmaengine framework : https://docs.kernel.org/driver-api/dmaengine/dmatest.html

Prior to this, you need to setup a DSA WQ of kernel type using accel-config.

For e.g.:

sudo accel-config config-engine dsa2/engine2.0 --group-id=0

sudo accel-config config-wq dsa2/wq2.0 --group-id=0 --block-on-fault=0 --mode=dedicated --priority=10 --wq-size=64 --type=kernel --name=dmaengine --driver-name=dmaengine

sudo accel-config enable-device dsa2

sudo accel-config enable-wq dsa2/wq2.0

Post this, there should be a dmachannel showing up under /sys/class/dma (it will not show up under /dev/dsa as it does for a user WQ).

Hope this helps.

[yun3319]

I tried executing the command suggested(sudo mode):
$ accel-config config-wq dsa2/wq2.0 --group-id=0 --block-on-fault=0 --mode=dedicated --priority=10 --wq-size=64 --type=kernel --name=dmaengine --driver-name=dmaengine

However, I encountered the following error message:
libaccfg: accfg_wq_set_str_name: wq2.0: write failed: Operation not supported

It seems that the operation is not supported when trying to configure the work queue with the specified parameters. Could you provide any insights on why this might be happening or suggest any further steps I could take to resolve this issue?

Thank you.

[RNa]

Are you doing it as a root user?

What is your kernel version?

[yun3319]

Yes, I did as a root user. And my kernel version is 5.18. Thank you.

[RNa]

How many DSA devices do you have on your system? lspci |grep -i 0b25 should tell you.

If you have 1 DSA per socket, you can try the earlier commands with dsa0 instead of dsa2, wq0.x instead of wq2.x and engine0.x instead of engine2.x.

[yun3319]

In my system, there’s eight dsa devices. And when I tried dsa0, wq0.0, I encountered same situation:
libaccfg: accfg_wq_set_str_name: [wq] write failed: Operation not supported

[fyu]

Is it possible that you configured and enabled wq0.0 or wq2.0 as user mode before you try to enable them as kernel mode without disabling them first?

You may check wq status by accel-config list. If wq0.0 or wq2.0 are enabled, please disable them before you change them to a different mode. You may run "sudo accel-config disable-wq dsa0" to disable dsa0 device including all enabled wq0.x. Then try the kernel wq configuration command.

[yun3319]

I was able to confirm that a dma channel(dma4chan0) was created in /sys/class/dma after disabling wq in linux kernel 6.9.0 and proceeding.

However, when I ran dmatest with following command, (all commands are executed by root user)

$ modprobe dmatest timeout=2000 iterations=1 channel=dma4chan0 run=1

it failed with the following messages:

dmatest: Added 1 threads using dma4chan0
dmatest: Started 1 threads using dma4chan0
idxd 0000:00:01.0: Device error 0x1e operation : 0x3 fault addr: 0x0
dmatest: dma4chan0-copy0: result #1: 'test timed out' with src_off=0x517 dst_off=0x104a len=0x20a2 (0)
dmatest: dma4chan0-copy0: summary 1 tests, 1 failures 0.49 iops 3 KB/s (0)

I configured device, engine, wq with the following command:

$ accel-config config-device dsa0
$ accel-config config-engine dsa0/engine0.0 --group-id=0
$ accel-config config-wq dsa0/wq0.0 --group-id=0 --block-on-fault=0 --mode=dedicated --priority=10 --wq-size=64 --type=kernel --name=dmaengine --driver-name=dmaengine
$ accel-config enable-device dsa0
$ accel-config enable-wq dsa0/wq0.0

Is there any problem with the configuration or dma test?

[RNa · Accepted solution]

Please check if VC1 is enabled in the device configuration (you can check via lspci output). 

If disabled, perhaps you can try to enable via setpci?

Alternately, you can specify tc_override module param when loading idxd driver via modprobe.

[yun3319]

Thank you for your kind reply. It was resolved by setting tc_override.