Arria10 DMA失败

[NZhan1]

使用10AX027和CPU调试PCIe时发现以下问题：

调试过程中软件驱动负责分配一块物理内存，然后把物理内存地址写入FPGA，FPGA从这个物理内存地址做DMA，读取内容。

在Xeon系列的硬件平台上的异常现象是：

当驱动分配的物理内存在4G以上时，FPGA做DMA是可以的，分配到4G以下时，FPGA发送tlp的读请求后，始终得不到响应，DMA失败。

做了以下尝试：​

1. 将FPGA及driver的DID和REV的register改掉，不为0x0000，DMA仍会失败。
2. FPGA将内部逻辑改为32位地址的模式，没有作用。
3. 当分配到4G以下的地址时，手动将高32位地址置1，DMA是可以的。
4. CPU修改BIOS的设置，没有作用。
5. 抓了AER数据，FPGA发送tlp的读请求后得不到响应时，没有抓到transpend+的数据​。

但是在ATOM C2758，酷睿 I5上分配4G以下的地址时，DMA都是OK的。

客户是自己写的DMA程序，请问可以判断问题可能出在哪里吗？

谢谢

[SengKok_L_Intel]

Hi,

As per what I understand here, this is using a custom driver to run the DMA test, where if the address allocation from the host is less than 4G, the test will fail.

Do you test the an829 design before?

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/an/an829.pdf

What is the "address width of accessible PCIe memory space" setting in the design?

Regards -SK

[NZhan1]

HI SK No, The DMA part is self-designed module. And use PCIE AVST Gen2x4 mode. But this DMA can work with other platform, like ATOM C2758 and Core I5. There is no such problem. Regards， Nicole

[SengKok_L_Intel]

Hi Nicole,

From the FPGA site, the PCIe is configured as AVST GEN2x4, then there is a custom module to connect the Avalon ST interface with the DMA? What is the memory size selected from the PCIe GUI's BAR tab?

When issue a memory write or read TLP from the Host, did you able to see the TLP appear at the Avalon ST interface?

If possible, please help to explain why is the DMA not working? Is this unable to receive the TLP from the Avalon ST interface of PCIe IP, or the PCIe IP sent incorrect packet to the DMA module in the FPGA?

Regards -SK

[NZhan1]

Hi SK BAR2 is enabled, type is 64-bit prefetchable memory, size is 1Mbyte-20bits. Currently, we send a TLP with memory read request, but can’t receive CPU’s completion. From the avst st interface, a read request is sent out. Regards， Nicole

[SengKok_L_Intel]

If the read request is sent out from the AVST to DMA. At this point, the PCIe IP still working fine. Do you mean the AVST interface does not receive the completion from the custom DMA? If yes, then this is required to debug from the DMA perspective.

Regards -SK

[NZhan1]

Hi SK， The read request is sent out from the DMA to AVST and then TX channels, and finally the CPU. But DMA didn’t receive the completion from the CPU. We will make sure that AVST can receive completion by signaltap first. It there any similar situation occurs on others’ Arria10 and Xeon CPU plartform before? Regards， Nicole

[SengKok_L_Intel]

Hi Nicole,

Thank you for the clarification. I don't aware if there is a similar issue. At this point, it does not look like the PCIe AVST problem yet. The PCIe AVST is a hard block, it will pass the TLP that received from the host to the DMA.

From the AER, does it show the completion timeout or any other error? If yes, it means that the Host does not return the completion.

If the completion package does not observe from the AVST interface, then you might need to debug it by using a PCIe protocol analyzer to confirm if the host really sent a valid completion TLP to the endpoint (FPGA).

Regards -SK

[NZhan1]

Hi SK For AER，what kind of errors should be monitor? Timeout error and? Is there a UG of AER errors definition can be refer to? Is there any organizations can provide access to PCIe protocol analyzer for normal customer? Thanks! Regards， Nicole

[SengKok_L_Intel]

For AER definition, you can refer to section 9.5:

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_a10_pcie_avst.pdf

If you are using Linux, the command below might be helpful to read the AER:

setpci -s B:D.F ECAP_AER+0x10.l

setpci -s B:D.F ECAP_AER+0x04.l

Regards -SK

[NZhan1]

Hi SK, So your suggestion is to check the two registers, and compare with “9.5. Uncorrectable and Correctable Error Status Bits” to locate the problem. The detailed status info will be found at PCI Express Base Specification. The “setpci -s B:D.F ECAP_AER+0x10.l” is used to check ”Correctable Error Status Register”. And the “setpci -s B:D.F ECAP_AER+0x04.l” is used to check” Uncorrectable Error Status Register”. Thanks! Regards， Nicole

[SengKok_L_Intel]

Yes, this is one of the debug item. Another thing is to capture the signal tap to confirm whether the PCIe HIP AVST interface did receive the completion TLP, this can help to understand whether the packet is missing.

Regards -SK

[NZhan1]

Hi SK , Thanks! We will try it. Regards， Nicole

[SengKok_L_Intel]

Please do let me know if you have an update or more help is needed here. Thanks.

[NZhan1]

Hi Sengkok, I have not got test result from customer, but currently there is no more requests. Thanks! ************************************************** DISCLAIMER: This email is for the use of the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We have taken precautions to minimize the risk of transmitting software viruses, but we advise you to carry out your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege. If you are the intended recipient and you do not wish to receive similar electronic messages from us in future then please respond to the sender to this effect. **************************************************

[SengKok_L_Intel]

Thank you.

[SengKok_L_Intel]

I will set this case to close-pending for now. Please do not hesitate to get back to me within the next 20-day close-pending period if more help is needed. 

Regards _SK