PCIe Hard IP Cyclone 10 peer to peer communication

[andrecap]

Hi,

we are trying to achieve communication between two endpoint FPGAs.

The architecture of the system is as follows:

                         PC (root complex)

                                    |

                           pcie switch

                         /                     \

               FPGA A               FPGA B

The FPGAs are using the PCIe Hard IP with DMA and Avalon-MM interface.

We can communicate from PC with FPGA A and B individually, both through the BARs and the fpga's pcie DMA.

When we program the fpgaA's DMA to make a transaction between A and B, like this:

- SRC: <on-chip-memory of fpgaA address>

- DST: <bar2 of fpgaB address + fpgaB on-chip-memory offset >

nothing happens.

We clearly see through signal tap that the packet is leaving fpgaA, but nothing exits from the PCIe rxm_bar to the avalon bus in fpgaB.

We also saw from the switch registers that the switch is correctly forwarding the packets.

What are we missing?

Why the FPGA B is not receiving any packet?

Kind Regards,

Andrea

[BoonT_Intel]

Hello Sir,

If you can see the data come out from the FPGA1, then usually the issue is due to the driver of the host. As you can see from this article, the p2p transaction is very dependent on the host.

https://www.kernel.org/doc/html/latest/driver-api/pci/p2pdma.html

and you can follow the recommendation to amend the host driver accordingly.

[andrecap]

Hello Mr. BoonT,

thanks for your rapid response.

I looked at the documentation, but that kind of discussion seems to be referring to the case where it's the host that is using the DMA controller (dmaengine in linux for something like that).

In our case, we are instead using the DMA controller inside one of the FPGA to orchestrate the transaction (for example, fpgaA).  I should have specified better.

We followed the documentation at this link to implement the DMA transfer (chap. 6.7.4):

https://www.intel.com/content/dam/www/programmable/us/en/pdfs/literature/ug/ug_a10_pcie_avmm_dma.pdf

Do you have any other suggestion?

Kind Regards,

Andrea

[BoonT_Intel]

Hi Andrea,

I check with our Intel PCIe team, we acknowledge that the p2p that you want to achieve is called the ACS (Access control services). I am sorry to say that the PCIe IP in the FPGA is unable to support this. Any transaction of the FPGA must go to the root complex directly. As you can see none of the Intel FPGA document did mention about the p2p, the IP is not designed to achieve this type of transaction. The AVMM DMA IP is just to transfer the data to other function component under the same Physical link like DDR4 within the same FPGA.

Anyway, my previous feedback is still valid, to achieve p2p, it is also depending on the Root complex can support it or not.

[andrecap]

Hi BoonT,

you're right, but I was probably expecting this to be explicitly specified in the documentation...

Anyway, may I ask which part of the IP is lacking support?

Is there a way I can workaround this problem (for example: adjusting the TLP packet by "hand")?

The perspective of rewriting the whole driver isn't very appealing, so I'd like to know if there are other ways to solve the problem.

Thanks.

Kind Regards,

Andrea

[BoonT_Intel]

Hi Andrea,

P2P is an optional specification from PCIe spec, and the IP is not developed with the consideration to support P2P. Thus, the IP does not claim that it can support P2P. And we never test the P2P using the IP. Thus, I can’t help much on this.

But just for sharing, I try to check the PCIe specification 3.0. Looks like to achieve P2P, we will need to trigger from the rp which is the host. From ep perspective, it is passive and just to ensure that it can send out the packet correctly with targeted address.

[BoonT_Intel]

We do not receive any response from you to the previous question/reply/answer that I have provided. This thread will be transitioned to community support. If you have a new question, feel free to open a new thread to get the support from Intel experts. Otherwise, the community users will continue to help you on this thread. Thank you.