Arria 10 - Avalon MM DMA for PCIe crashes the PC when re-initializing the descriptor controller

[ThomasKindler]

Hi!

I'm using the Avalon MM DMA for PCIe on a Terasic TR10a-HL Arria 10 board.

The card is running the provided PCIe_Fundamental.sof demo image and is plugged into a server PC.

I'm currently writing a VFIO based linux userspace driver for the card.

As outlined in ug_a10_pcie_avmm_dma-683425-666332.pdf, my program allocates DMA memory, and initializes the Read and Write DMA descriptor controller registers via BAR0.

On the first program run, everything works fine: I can issue read and write DMA requests.

Problem: If I start the program a 2nd time and re-initialize the descriptor controller with different descriptor table addresses, the PC reboots.

Seemingly, the FPGA issues some random DMA requests that crash the PC.

Workaround: Reconfigure the FPGA before each program run. This will reset the descriptor controller to its reset state (RD_DMA_LAST_PTR = 0xFF, WR_TABLE_SIZE = 0x7F, everything else 0).

Questions:

1. What is the correct sequence to re-init the descriptor controller?

The ug_a10_pcie_avmm_dma manual says on page 82:

RC Read Status and Descriptor Base (Low)

[..] To change the RC Read Status and
Descriptor Base (Low)base address, all
descriptors specified by the RD_TABLE_SIZE
must be exhausted.

What does "exhausted" mean in this context?

2. Is there a way to reset the DMA engine without reconfiguring the FPGA?

[ThomasKindler]

Update:

As I learned from another post https://community.intel.com/t5/FPGA-Intellectual-Property/How-to-stop-or-cancel-an-Read-Write-AVMM-DMA-operation-after-it/m-p/1288312,

"One possible solution is to generate the Secondary Bus Reset from the host. It is available at the Bridge Control register Bit 6".

This indeed resets the DMA descriptor controller registers, and I could use it as a possible workaround.

The control register resides in the bridge before the actual device:

$ lspci -d 1172:e003 -PP
40:03.1/41:00.0 Non-VGA unclassified device: Altera Corporation Device e003

$ sudo setpci -s40:03.1 BRIDGE_CONTROL
0012
$ sudo setpci -s40:03.1 BRIDGE_CONTROL=0052
$ sudo setpci -s40:03.1 BRIDGE_CONTROL=0012

Still a reset bit on the card itself, or a non-crashing initialization sequence would be preferred!

[wchiah]

Hi,

Thank you for reaching out.

Just to let you know that Intel has received your support request and I am assigned to work on it.

Is it the post mentioned in your second reply able to help you to solve your problem ?

Best regards,

Wei Chuan

[ThomasKindler]

Thanks for looking into it!

As said, the hot reset is only a workaround - it takes ~1s, and as I understand, may reset other devices connected to the same bridge.

So an official sequence to reprogram the descriptor controller's descriptor table registers would be preferred.

[wchiah]

Hi,

For the descriptor controller table, you may refer to link below

https://www.intel.com/content/www/us/en/docs/programmable/683425/18-0/write-dma-descriptor-controller-registers.html

Hope this able to help you.

Regards,

Wincent_Intel

[ThomasKindler]

Yes, those are the registers I'm talking about. It's the same as the "ug_a10_pcie_avmm_dma-683425-666332.pdf" document that I referenced in my first post.

My program initializes those registers and can issue read and write requests according to the example:

https://www.intel.com/content/www/us/en/docs/programmable/683425/18-0/read-dma-example.html

But I can't initialize those registers for a second time.

If I rewrite the descriptor controller registers, the PC crashes.

My question was:

How can I reset or deactivate the DMA descriptor controller before rewriting the registers?

[wchiah · Accepted solution]

Hi,

The pin_perst is the power on reset of the FPGA board, when you reboot the host, it will toggle the pin_perst. Where the npor is driven by soft reset (user application), but the soft reset will also bring down the link, once link being bring down, we need to rebooting the PC.

I have a discussion with another PCIe specialty, it seem like there is no other alternative to reset the DMA controller alone beside reboot the PC.  

Hope this answered your question.

Regards,

Wincent_Intel

[ThomasKindler]

Sad to hear this, but OK.

We are then trying to extend the generated descriptor controller code by a new RESET bit in the control register.

If that doesn't work out for some reason, we can continue to use one of the possible work-arounds:

1. Use a fixed IOMMU mapping, so the descriptor addresses stay the same
2. or issue a PCIe hot reset before starting the program
3. or re-Configure the FPGA before starting the program
4. or use a kernel driver instead of UIO/VFIO

[wchiah]

Hi,

Thanks for sharing with me some of your thoughts.

Is there anything else I can help you ? Else I would like to have your permission to close this forum case.

Regards,

Wincent_Intel

[wchiah]

Hi

We have not hear from you and this Case is idling. It is not recommended to idle for too long.

Therefore following our support policy, I have to put this case in close status. My apologies if any inconvenience cause

Hence, This thread will be transitioned to community support.

If you have a new question, feel free to open a new thread to get support from Intel experts.

Otherwise, the community users will continue to help you on this thread. Thank you

If you feel your support experience was less than a 9 or 10,

please allow me to correct it before closing or let me know the cause so that I may improve your future support experience.

Regards,

Wincent_Intel