Point me in the right Direction - PCIe

[Altera_Forum]

Ok - Ive been staring at the PCIe Userguide for a couple of days - and I dont really know where to start with this thing. We have a design that requires the following setup: 

 PC (currently windows, later Linux) <-- PCIe x1 --> Arria2GX190 16k Shared ram <-- Control Interface --> Arm Chip  Basically the Arria 2 just acts as a bridge between the Arm and the PC. Dont really know where to start with the PCIe core. Very little experience with PCI. Anyone know some design to get me going. Some nice demos of the core?

[Altera_Forum]

It is also worth squeezing as much info as possible into the words that are read by PIO. 

 I did think about making the 'interrupt pending' register (1 bit per interrupt source) be 'read and clear' instead of using a write to clear the bits down. But that causes problem when running a diagnostic program to dump the register area!  Our fpga is directly connected to a PPC on the same card. I carefully worked the software interface between the two parts to minimise the number of separate PCIe transfers - in our case the fpga is always a slave (I got 16M SDRAM put down for buffers, everything else is internal).  I did have to get the ppc to use dma (embedded in the PCIe controller of the ppc we are using - no linux support) in order to get adequate throughput. Since a 128 byte dma transfer (1 PCIe request) doesn't take much longer than a single cycle, the driver just spins waiting for the tranfser to complete (actually they are double bufferred with copies to/from userspace).  For the medium speed IO this was fast enough and doesn't slug the ppc that much (this is part of why we have a ppc).

[Altera_Forum]

Oliver, 

 Have you looked at the SGDMA demo design? Its register set is fully documented and it supports a generic bidirectional descriptor-table-based DMA approach. The sad thing about the SGDMA example code is that it is not fully PCIe compliant (e.g, no completion timeout mechanism), ugly code in my eyes, and you will have a hard time tailoring hardware and driver to your specific needs if the base code is not enough.  I agree, it would be great if there was knowledge provided on how to craft a DMA design from scratch. Truth is, every design is different on its internal hardware interfaces as well as the required software interfaces. Is it supposed to act as a network adapter? Does it transport a stream of data? Or multiple such streams? Is there an internal CPU to interface with, or just hardware? Is there a memory or a Fifo to interface with? Do you need support for auxiliary functions like frame time stamping, hold-off, shared memory, flow control, other functions … ?  – Matthias

[Altera_Forum]

Oliver, 

 I recently shipped a PCIe design using Altera's Cyclone IV GX hard IP core. See the link below for the ref des that I used. Altera has several ref des for PCIe but I chose this one because I think it's rather intuitive and easier to add your custom logic to it. The DMA is not scatter-gather (it requires contiguous space) but this can serve as a baseline for your PCIe DMA design. Hope this help.  http://sites.google.com/site/ednalabs/project/fpga/tips/pcie-design-with-altera      --- Quote Start ---  I'm some of the way through a PCIe design with a Cyclone IV. So far I've learned:  A. Start with a working design from a dev kit or eval kit. That way you start with known working hardware, FPGA design and windows driver. If you don't start from there you will never know if your problem is with the FPGA, hardware or windows driver.  B. Altera's eval kits break easily. The FPGA design only seems to get tested on the version of Quartus that was current when the design was done. Later versions of Quartus may break it! The reference designs from MEV are much better.  C. Using an Avalon-MM based solution and SOPC builder works. Unfortunately, as others have warned, it's really slow!  D. Don't bother with getting the PCIe specs until you know what you don't know.  Does anyone know an easy route to a DMA solution?  --- Quote End ---  

[Altera_Forum]

Thanks for the links. I think I'm going to bury myself away for a while to study all this stuff!

[Altera_Forum]

--- Quote Start ---  Matthias, I've just reached the point where it's obvious that our design must use DMA. This is a significant setback as writing a custom DMA driver is going to impact our timescales quite badly. My knowledge of PCIe transactions is negligible, so we are looking for additional expertise in this area. It's a shame that there is no generic solution from Altera and that every user must make their own solution from scratch. Oliver  --- Quote End ---     Oliver, you said "Unfortunately, as others have warned, it's really slow!".   Is it fast enough?   Once you get your head round the MM address translation tables DMA functionality with the MM interface is fairly straight forward.     Nial.

[Altera_Forum]

--- Quote Start ---  Oliver,  I recently shipped a PCIe design using Altera's Cyclone IV GX hard IP core. See the link below for the ref des that I used. Altera has several ref des for PCIe but I chose this one because I think it's rather intuitive and easier to add your custom logic to it. The DMA is not scatter-gather (it requires contiguous space) but this can serve as a baseline for your PCIe DMA design. Hope this help.  http://sites.google.com/site/ednalabs/project/fpga/tips/pcie-design-with-altera  --- Quote End ---    That site links to this reference design (AN575) http://www.altera.com/support/refdesigns/sys-sol/indust_mil/ref-pciexpress-ddr2-sdram.html  However there are no download links to get the source on that page, other than the user guide. Where do you get the actual reference design?  I found this one as well (AN431): http://www.altera.com/support/refdesigns/ip/interface/ref-pciexpress-ddr3-sdram.html  Is that just an updated version of AN575? I can't even get it to compile properly.  In any case, I'm trying to find a reference design in VHDL rather than verilog if possible. It's starting to become pretty frustrating to find a starting point for my PCIe project. I'm just trying to dump data from custom logic over PCIe to a solid state drive - no NIOS/Qsys/OS apps involved.  -Ian

[Altera_Forum]

Ian, 

 Here's the link for downloading the PCIe reference designs including the one I specified. The one I specified is not the newest or most recently written PCIe reference design from Altera but I just feel like it's the easiest one to re-use given the way it was written/designed. Not all Altera's PCIe ref des are written or designed in the same style because they were designed by different engineers so you will have to pay attention and choose which one you can best follow/re-use. Hope this help.  https://www.altera.com/support/software/download/refdesigns/ip/interface/dnl-pciexpress-ddr3-sdram.jsp      --- Quote Start ---  That site links to this reference design (AN575) http://www.altera.com/support/refdesigns/sys-sol/indust_mil/ref-pciexpress-ddr2-sdram.html  However there are no download links to get the source on that page, other than the user guide. Where do you get the actual reference design?  I found this one as well (AN431): http://www.altera.com/support/refdesigns/ip/interface/ref-pciexpress-ddr3-sdram.html  Is that just an updated version of AN575? I can't even get it to compile properly.  In any case, I'm trying to find a reference design in VHDL rather than verilog if possible. It's starting to become pretty frustrating to find a starting point for my PCIe project. I'm just trying to dump data from custom logic over PCIe to a solid state drive - no NIOS/Qsys/OS apps involved.  -Ian  --- Quote End ---  

[Altera_Forum]

Few basic questions: 

1. How many functions can be implemented in PCIe hard IP in End Point Mode? 
2. Does this IP support ARI? 

[Altera_Forum]

We have 4 BAR defined (0, 2, 3 and 4). 

The last one uses the PCIe to Avalon master bridge and so can access a lot of Avalon slaves. (Don't ask why some of the others are separate!) You do have to be careful about how the PCIe slave window in mapped into Avalon address space, there are some strange assumptions made - it some cases it might be necessary to feed the Avalon master signals through a 'conduit' to generate the correct, fixed, upper address bits.

[Altera_Forum]

--- Quote Start ---  Oliver,  I recently shipped a PCIe design using Altera's Cyclone IV GX hard IP core. See the link below for the ref des that I used. Altera has several ref des for PCIe but I chose this one because I think it's rather intuitive and easier to add your custom logic to it. The DMA is not scatter-gather (it requires contiguous space) but this can serve as a baseline for your PCIe DMA design. Hope this help.  http://sites.google.com/site/ednalabs/project/fpga/tips/pcie-design-with-altera  --- Quote End ---   Hello, alybruin! Are You sure, that AN 431: PCI Express to External Memory Reference Design has only simple DMA ( it requires contiguous space) without scatter-gather? On page 8 of AN431 it is said: "The Qsys system contains PCIe, OnChipMem, DDR3, and msgdma."