PCIe DMA transfer timeout

[allen18]

Hello，

    I am using quartus Pro 22.3 , device ( stratix10 1SX110HN2F43I2VG ) ,  IP core ( L-Tile and H-Tile Avalon Memory-mapped Intel FPGA IP for PCI Express 22.2.0 .

    When I use the DMA IP core to transfer data to CPU, I always encounter DMA timeout issues, occurring every few seconds.  The kernel prints “DMA operation timed out”.  

    Even if I generate the example design and follow the steps in the UG, I still encounter the same issue. ( enter 0 for run DMA , enter 0 for infinite loop, DMA operation timed out after few seconds).

    How should I address this issue ?

Thanks.

 

[ventt]

Hi @allen18,

Thanks for reaching out.

Allow me some time to investigate your issue. I shall come back to you with the findings.

Thanks.

Best Regards,

Ven

*Kindly note that 8 July 2024 is a national public holiday. Please expect a delay in response.

[ventt]

Hi @allen18,

May I check with you on the following items?:

1. Can you please check if the PCIe linked up successfully when the error occurred?

2. Are you using an Altera development kit or a custom board?

3. Could you please share the .ip file, so that I can generate the Example Design based on the parameters you specified?

4. Is this a root port or an endpoint design?

5. Are you using the 'DMA' Example Design?

6. Which PCIe Gen is used in the Example Design?

Thanks.

Best Regards,

Ven

[allen18]

  Hi @ventt ,

  I'm not sure if the DMA timeout is due to the PCIe link down. I suspect it's not.

  I'm using a custom board,endpoint design, the relevant IP files are attached. The platform design file is simply the example design.

[ventt]

Hi @allen18,

Thanks for attaching the files.

Could you please try to run the DMA test using S10 Dev Kit following the FPGA Wiki? This checks if the environment you are using is compatible with the example design.

Reference Design: Gen3x8 AVMM DMA - Stratix 10:

https://community.intel.com/t5/FPGA-Wiki/Reference-Design-Gen3x8-AVMM-DMA-Stratix-10/ta-p/735907

Another thing, since you're using a custom board, could you please confirm that the correct pin assignments have been made before compiling the project?

Thanks.

Best Regards,

Ven

[ventt]

Hi @allen18,

Do you have any updates?

Thanks.

Best Regards,

Ven

[allen18]

Hi @ventt ,

My pin assignments should be correct because PCIe timeouts occur after running for a while. I don't have a development board for verification, but I believe this issue is related to the PCIe driver rather than hardware. 

Thanks.

[ventt]

Hi @allen18,

Could you please file an Intel® Premier Support (IPS) case to enable us to further analyze and debug your issue?

You can do this by reaching out to your Distributor Field Application Engineer (DFAE), or you may follow the steps outlined in the KDB provided below.

KDB: https://www.intel.com/content/www/us/en/support/programmable/articles/000086520.html

Thanks.

Best Regards,

Ven

[ventt]

Hi @allen18,

Meanwhile, I will try to run the DMA test on my end to see if I can replicate your issue. Could you please address the following questions, and also please see the suggestions below?

1. Please provide the 'lspci' information.

2. Are you seeing the error in the latest version of QPP software, v24.2?

3. Please check the OS and the version used. We suggest you follow the OS requirement stated in ug-683616: https://www.intel.com/content/www/us/en/docs/programmable/683616/20-3/hardware-and-software-requirements.html

4. When running the DMA test, we suggest the CPU not run other tasks.

Thanks.

Best Regards,

Ven

[allen18]

hi @ventt ,

  You can use the corresponding example design in your test environment to run DMA and verify if timeout issues occur. Our driver engineers have developed new kernel driver code, and with the new driver, we no longer encounter DMA timeout issues during data transmission. This confirms that the problem is definitely related to the test kernel code provided by the example design,or possibly to some definitions in the header files of the code.

[ventt]

Hi @allen18,

I was able to run the DMA test for over 50,000 iterations in my environment without encountering any DMA timeout issues, as shown in the attachment. The operating system used is CentOS 7. The software drivers generated from the design examples in both QPP v22.3 and v24.2 (the latest version) were tested and did not run into errors when performing the DMA test in an infinite loop.

May I ask which operating system you are using in your environment? Please ensure that it meets the requirements specified in the user guide, as the driver has been tested and verified with the OS.

Thanks.

Best Regards,

Ven

[allen18]

These are my system information，perhaps environmental differences are causing these kinds of issues ?

[ventt]

Hi @allen18,

Yes, I agree with you. The software driver generated with the design example has been tested in the environment (OS version etc.) as specified in the UG. If a different environment is used, there may be issues when running the test, and modifications to the code might be necessary.

With that said, do you have any further questions in this thread? If not, I will transition the thread to community support.

Thanks.

Best Regards,

Ven

[ventt]

Hi @allen18,

Since there are no further inquiries in this thread, I now transition this thread to community support. If you have a new question, please login to https://supporttickets.intel.com/, view details of the desire request, and post a feed or response within the next 15 days to allow me to continue to support you. After 15 days, this thread will transition to community support. The community users will be able to help you with your follow-up questions.

Thanks.

Best Regards,

Ven