Hi @wchiah ,
I am recreating this question because the older one was closed. There is a problem with notifications. I am not getting notifications of replies to my posts. How can I correct this?
To get to the problem, The MCDMA/PCIE core has 11 bits of DMA channel address when using multiple logical channels on one physical interface. We planned to use 256 channels (physically) but probably only a dozen or so logical channels. However, when we try to drive it using Intel's DPDK, we seem to get corruption of the D2H DMA descriptors if we use more than 64 channels. Why are we seeing this limitation? Is it in the DPDK itself? Also, it seems we cannot use logical channels that we can map to a physical channel address. It seems that the creation of channels automatically uses sequential physical channel addresses starting from 0. Is it not possible to map logical channels to physical channels?
To answer your question, we are using Quartus 21.2 for Agilex AGFB014 (P+E Tile).
In our build settings, we have:
PCIe0 Settings/PCIe0 IP Settings/PCIe0 PCI Express/PCI Capabilities/PCIe0 Device/PF0 = 256; and PCIe0 Settings/PCIe0 IP Settings/MCDMA Settings/D2H Prefetch channels = 256; and Maximum Descriptor Fetch = 16
Thanks for reaching again.
- Do you try to run the example design without modification ? is it behave the same ?
- What kind of payload size you are using to do the test
- Did you develop your own driver or you are using our example design driver ?
- Did you capture the below signals to check if the data is mismatch or not ?
Looking forward to hear back from you.
Hi @wchiah , thanks for the reply,
we used the example design originally without modification and it was working, with both example driver and our driver.
Then we removed the packet generator/checker from the example to create our own design which also worked with both our driver and example design driver.
Then we tried to modify the MCDMA/PCIE core to 256 channels from 64, and this is where we start having problems. We used payloads of 8K for the 64 channel case, and 2K for the 256 channel case.
Why does the example design only use 64 channels? Why not 256 or the max 2K?
We are now in the process of trying the example design customized for 256 channels. We will try to get reports from the example driver to you.
We do monitor those signals in our design with signaltap but do not yet have a capture when it fails, but we'll keep trying.
Hi @grspbr ,
If refer to the user guide, there is a 1.2. Known Issues
Where if setting Multichannel D2H AVST configuration has stability issues when total number of D2H channels configured is greater than 256. Please consider this in your design.
Also, if refer to 188.8.131.52. Avalon-ST 1-Port Mode
- In the current Intel® Quartus® Prime release, the D2H Prefetch Channels follows the total number of DMA channels that you select up to 256 total channels.
- When the total number of channels selected is greater than 256, then D2H Prefetch channels are fixed to 64.
- The resource utilization shall increase with the number of D2H prefetch channels.
- For details about these parameters, refer to the 4.12.2. D2H Data Mover Interface
Hope this answer your question.
Note that we are not exceeding 256 channels. We are trying to use more than 64 channels though. When we try 65 channels, for example, we get a hang up in the PC. We think that the 64 limit on descriptor prefetch may not be handled properly, either in our code or the example driver so we are looking at that right now.
The documentation is lacking here. There are 2K physical channels - i.e. 11 bits of physical address on the core for channel ID - yet we can only build a MCDMA with 256 channels, and yet again we only have 64 descriptor prefetch buffers. We don't find any documentation on how to address the 256 channels with 64 prefetch buffers. There needs to be a mapping of the prefetch buffer to a physical channel but we find no documentation for this in the software or hardware references. So how do we use 256 channels? We want this because we have a 256 channel AVST interconnect in our design.
If refer to the user guide, the D2H is limit to 64 channel and not available be set to more than that.
- If refer to your first question,
"However, when we try to drive it using Intel's DPDK, we seem to get corruption of the D2H DMA descriptors if we use more than 64 channels. "
- You are trying to set the D2H more than 64 channels and it is fail.
- Or you are trying to use 256 MCDMA channel ? Appreciate if you can describe more about it.
- Can I know which version of quartus you are using ?
- Can you please update your Quartus version into latest 22.3 or 22.3 and see if the same problem still happen?
Hoping to hear back from you.
I think you are misreading the guide... it says if you select GREATER than 256 channels, THEN prefetch channels is limited to 64. However, we are selecting EXACTLY 256 channels. Further, I may have mispoke in previous post, but we have also selected 256 Prefetch channels (not 64 as we once did). So we have 256 channels and 256 prefetch channels which gave no errors in Platform Designer and compiled successfully in Quartus 21.2. Or problem occurs when we try to configure channels above 64 using Intel's driver. Our software guy is checking the code again for possible roll-over in the channel numbering. Honestly, this looks like a software/driver problem to me.
Our software guy believes he has found a bug in the MCDMA core. By reading the prefetch channel QSR info, the start_address and consumed_address have experienced a roll-over in the configured address - i.e. does not match programmed linearly increasing address. I will try to conpile the design with the latest compiler and see if the problem persists.
Thanks for providing those detail, please allow me to have sometime to investigate on it.
Meanwhile, can I get your help to try to generate a new design example from your current Quartus v21.2 to the latest v22.3 or 22.4 and see if the problem able to replicate?
Attached is the qar file of the example design we just tried, from Quartus 22.4. It also failed when trying to use more than 64 channels. I'll send the qar file of our quartus 21.2 version as well.
I could not attach the file as its 78 MB and the max is 71MB
maybe you have ftp I can use?