API calls failed while running PCIe DMA transfer example design

[Sijith]

Hi,

I was working the PCIe DMA transfer example design for Arria 10. I have added a counter custom IP which counts upto 1000, and An Avalon FIFO IP with the design. My intention is to write the data created in counter to DDR4 and then use the DMA API call (provided by Terasic in Demonstrations/PCIe_SW_KIT/Windows/PCIe_DDR4/PCIE_DDR4.cpp) to read them from PC.

Even if I follows the steps given in the User manual of DE5a_DDR4_NET (attaching the manual. The chapter-7 section 7.6 is what I was following), the DMA API calls to read data fails. I would like to know why I could not run Read DMA API from the PC.

The It would be great if someone could give any helping hand. I am attaching the error message (Crash_op.PNG). 

I am just listing the procedure I followed,

1. Installed both DDR4 2400 4GB SODIMM on the FPGA board. 

2. Connected the FPGA board with PC through PCIe. 

3. Configured FPGA with DE5A_NET.sof (here the design .sof having PCIe DMA transfer example design + Avalon FIFO + Counter custom IP) by executing the test.bat. 

4. Restart Windows 

4. I could see the PCIe driver in the device manager (Windows has detected the FPGA Board). 

7. Executed the PCIE_DDR4.exe. Then  in the menu putting the options 3, 4 and 5 gives the failure.

FYI: I have been using Quartus Prime Pro 18.1 in Windows 10. 

[wchiah]

Hi,

Theoretically, DMA function is not work for FIFOs but it is only work for memory that have physical address. FIFO unlike a memory (DDR3) and no address associate with it. Thus, for FIFO, we usually used write/read32.

Regards,

Wincent_Intel

[Sijith]

Hi,

Sorry If my question was not clear. Actually, I my project is to stream data generated in a custom IP (a counter here)  to a FIFO (Avalon FIFO IP) using an Avalon streaming interface. Then this data in FIFO should go into the DDR4 memory. The data thus written at the DDR4 should undergo DMA transfer to the host PC through PCIe.

For this, I was using the default PCIe DMA transfer example design (that given by the Terasic)  and I have added the custom IP and Avalon FIFO IP with the PCIe DMA transfer example design (I can send the design files if you would like to see how they are connected in Platform Designer System).  

In the API code (API in the PCIe_DDR4.cpp provided by the Terasic, I did comment out the data generation part and the DMA write portion. I just retained the DMA read portion and trying to print the *pRead values (Please see the lines 246 of the PCIe_DDR4.cpp files attached) that is supposed to fill by the data from the FIFO. Also attaching the verilog counter code used for the custom IP generation.

Just curious that any reason for the API call failure (that evident from my explanation)?, If so please give recommendation/suggestion to go forward? 

Please let me know if you have any question.

[wchiah]

Hi Sijith,

To be honest, I does not try this before.

I check the track record, I does not see any same implementation before as well.

However I can lay down some of the suggestion for you to proceed further.

1. Verify the connection and configuration: Ensure that the connections between your custom IP, Avalon FIFO IP, PCIe DMA IP, and DDR4 memory are correctly established and properly configured. Check the signal interfaces, clock domains, data widths, and address mappings to ensure consistency and correctness.
2. Debugging the API call failure: If the API call is failing to retrieve the expected data from the FIFO, it's essential to debug the issue step by step. You can add debug print statements or utilize simulation tools to trace the flow of data from the Avalon FIFO IP to the PCIe DMA IP and then to the host PC. Monitor the status and signals at each stage to identify any potential issues, such as incorrect pointer assignments, data alignment problems, or synchronization issues between different clock domains.
3. Verify the correctness of the DMA read process: Make sure that the DMA read process, specifically the part you retained from the example code, is correctly configured and operates as expected. Check the pointer assignments, buffer sizes, and any required memory mappings to ensure that the DMA read process can access the correct data from the DDR4 memory.
4. Analyze error codes and messages: If the API call is returning an error code or generating error messages, analyze them to understand the specific issue or failure reason. Cross-reference the error codes with the documentation provided by the Terasic example design to gain insights into the potential causes of the problem.

Hope this can help you.

Regards,

Wincent_Intel

[wchiah]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

[Sijith]

Hi Wincent,

I am very sorry for a bit delay in response. I was in a short vacation, and did not have internet access much of the time. 

As I was working in this project from last March, and was in a communication with an intel employee. He tried to help me in debugging and he was kind enough to have a couple of meeting with me to fix the issue. But he told me that the issue looks out of his expertise and he believed that the issue is somewhat related to the PCIe DMA part of the modified design (I mean adding the counter FIFO somehow alters the the DMA transfer example design), But he could not pin-point this. He suggested me to open a new thread with PCIe tag. So I would really appreciate if you could give me directions to solve this issue. 

I have a couple of questions regarding your suggestions,

1. "Verify the connection and configuration: Ensure that the connections between your custom IP, Avalon FIFO IP, PCIe DMA IP, and DDR4 memory are correctly established and properly configured. Check the signal interfaces, clock domains, data widths, and address mappings to ensure consistency and correctness."  

 As I am comparatively new to Quartus Prime Pro, I would like to have bit more details regarding this. I wonder is there any specific place to cross-check all at a place like compilation report (message)? . Or have to check them starting from the stage where we add them in the Platform Designer System.

      2. "Debugging the API call failure". 

I would like to know that I have simulated the counter+FIFO stage and simulation looks fine (I have used modelsim free version for that) But unfortunately I did not find a way to simulate the whole design (counter + FIFO + PCIe DMA transfer example). As the modified PCIe  DMA transfer example design has Avalon bus and its inherent complexity, I found its not as simple as simulating a combination of IP's by executing the simulation script (https://www.youtube.com/watch?v=eviC0jP90ZA&t=11s). Here my main issue was  get the top level simulation test bench file for the whole design and I was confused how to do it for the PCIe IP?.  Do you have any suggestion for doing the simulation of the whole design?

Debug print statements you mean in the API script?

      3. "Verify the correctness of the DMA read process"

 Any suggestion from your side to do this?

     4. "Analyze error codes and messages" :

  For sure its in my immediate todo list and I will update you the result soon. But I am very sorry to say I found Terasic document is not that helpful for the beginners like me and it does not have much information in it.

Thank you very much and looking forward to hearing from you. 

[wchiah]

Hi,

FPGA programming is quite a complicated tools.

Where some of the simple pin/assignment or parameter missmatch will cause the compilation fail.

To be honest, I do not have a direct solution to the problem that you are facing.

As a beginner, The simple way I would suggest you try the PCIe DMA design example.

https://www.intel.com/content/www/us/en/design-example/714463/arria-10-an708-pcie-gen3x8-avmm-dma-with-external-memory.html

From there you can try to understand how it work, then proceed to modified.

Hope this able to help you move forward.

Regards,

Wincent_Intel

[Sijith]

Hi,

I don't believe its a compilation failure, as the counter + FIFO + DMA transfer design are compiling without any error. I would like to report that now I am not getting the API failure that I mentioned before. I could resolve that by the driver re-installation.

But I still not able to get  the  counter data (at the DDR4) that I am creating in the counter. A video of output is attached. the data I am expecting at the DDR4 is the counter output that counts from 0. It would be great if you could have a look to see what can cause this. If needed I can give access to my working directory to you.

Thank you.

[wchiah]

Hi Sijith,

Since like we might have a communication barrier in understanding the question.

Would you mind if we can have a Teams meeting on this to better address this ?

Regards,

Wincent_Intel

[wchiah]

Hi,

I wish to follow up with you about this IPS case.

Hoping to hear back from you so that we can proceed for next step.

Regards,

Wei Chuan

[Sijith]

Hi,

Sorry for bit delay in responding as I was out from office. It would be great if we could arrange a google team meet on early next week? Monday or Tuesday? Do you have any time preference? Great if we could communicate further regarding the meeting through email? Thank you.

[wchiah]

Hi Sijith,

You may try the to simulate the whole design by referrring to

2.2 Simulating the Example Design in ModelSim

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/archives/ug-a10-pcie-avmm-dma-16.1.1.pdf

or you may try this document as well

https://www.intel.com/content/www/us/en/docs/programmable/683305/19-4/simulation-quick-start.html

Regards.

Wincent_Intel

[wchiah]

Hi,

Do you try the simulation tools ?

Is it work for you ?

Regards,

Wincent_Intel

[Sijith]

Hi Wincent,

I am still working on it and will update you. I am just curious that will you be available for a video meeting on tomorrow? Actually my colleague who is also working to debug have some questions. As he is in the USA, early morning singapore time will be good. Thank you.  

[wchiah]

Hi,

Sorry I miss out your message. Maybe can you sent your question here ?

then i can try to look at it before we have any meeting.

Regards,

Wincent_Intel

[wchiah]

Hi,

I wish to follow up with you about this case.

Do you have any further questions on this matter ?

​​​​​​​Else I would like to have your permission to close this forum ticket

Regards,

Wincent_Intel

[Sijith]

Hi Wincent,

I really would like to have your help in debugging the design issue I am facing that is not resolved yet. After  a week off, I just back to work, so sorry for the delay in response. I will post the result of the simulation of the design. Thank you very much.

[wchiah]

Hi Sijith,

Any update on the simulation waveform ?

Regards,

Wincent_Intel

[treble99]

Hello Wincent,

I am also working with Sijith on this. I am trying to simulate the ep_g3x8_avmm256_integrated design in modelsim. I have referred to PCIe Solutions User Guide section 2.2 and am able to run the tcl script after some minor troubleshooting.

I am able to view the waveform but I do not see much useful data there. I am not able drive the signals.

I am currently referring to the second link i.e. modelsim quickstart.

I will get back to you with some concrete questions soon.

Thank you!

[treble99]

Hello Wincent,

 I read through both documents but some things are still not clear. I also do not have experience in simulating BFMs before.

1. I see a set of BFMs but How do i drive signals? I want to verify that data is being read and written to each DDR a and b.

    what should I add in The generated tcl script  msim_setup.tcl

2. I read the solution guide and it talks about receiving a successful message after `run -all`, I let it run for a really long time and I do not get any message and my log file is also empty.

Attaching screenshots.

Appreciate the help!

Thank you,

Manish

[wchiah]

Hi Manish,

For your first question on driving the BFM,

Maybe you can try https://community.intel.com/t5/Intel-Quartus-Prime-Software/How-to-use-Avalon-MM-Master-BFM/td-p/170525

Let me know if you still unable to get it.

Regards,

Wincent_Intel