How to connect more than one FPGA-board

[Altera_Forum]

Hi, 

 I am trying to educate myself a little on connecting three or more FPGA-boards.   to make it short:  If any of you guys sit on information/resources that you would care to share regarding connecting FPGA-boards, it would be very much appreciated. I realize there's alot of questions asked in this post, but any input is appreciated and of interest. Any information that will help me progress would be of value.     intention/goal: What I am trying to accomplish, is to have one or more boards process data and then send them on to one "main"-board which will then further refine the results from the other cards.      connecting boards: For now I am looking at three terasic DE4 Development and Education Boards. (Regrettably I can't post links, since this is my first post on these forums ).   I seem to find myself at a loss finding information about how to connect them together.   Can two boards connect through the HSMC interface? Alternatively, if the boards were plugged into a PC motherboard, would they be able to communicate over the PCI-bus? Which is simpler? Other alternatives? Would other solutions offer more bandwidth?   Are there perhaps other boards more suited for the purpose?     software: Also, once the hardware setup is in place. Do Altera tools provide support for managing more than one board? Or must any communication between the boards be handled on a boardlevel/from scratch?    I suppose what I am trying to ask is: is it complicated addressing resources on one board ( eg. memory ) from another board in such a setup. How does the Quartus II software ( for instance ) handle more than one set of resources/boards. Multiple projects?   Are there other software tools specifically suited for the purpose perhaps?     documentation: Is there any documentation on these matters? I looked, but wasn't able to find much information about fpga-clustering on the Altera site ( or others for that matter).     budget: The sky is the limit. Well not totally, but the economy of the solution is of less interest at the moment.        Thanks, A.

[Altera_Forum]

Hi,  

they can comunicate to each other over the PCIe bus. Since the DE4 has a 4x PCIe 2.0 interface, you can get 4x 5 Gbit/s full duplex for each DE4, assuming the motherboard supports it.  It's also possible to use the HSMC connector. You'll need to find or design a cable or PCB with suitable pinouts to interconnect two DE4s. Maybe SAMTEC has suitable assemblies.  http://www.samtec.com/flex_circuitry/flex_standard_hs.aspx?s=cafc  In terms of software and design, there's no support for managing more than one FGPA.  Of course, Altera provides multiple comunication blocks (PCIe, SeriaLite, ALT_LVDS, etc) that you can and should use but beyond that, you're on your own.

[Altera_Forum]

You can use Samtec cables for connecting boards via the HSMC connectors, however, I would not recommend it. I have the cables, and there is no mechanical retention mechanism on the cable, so in my opinion it would not be very reliable in a system.  

 You could connect multiple boards via their PCIe interfaces. I have a board from OneStopSystems;  http://www.onestopsystems.com/pcie_atx_bp.php  that I plan on using for testing a couple of Stratix IV GX development kits. The board connects to the PCI-Express card on my laptop. A nice convenient setup.  What are your data flow requirements? Where is the data coming from? Are you talking MB/s, GB/s, TB/s? Or is the task 'processing' dominated?  Do your FPGAs need to be reconfigured dynamically, or can they be configured once at power-on? This is a critical question, as reconfiguring a device on the PCIe bus would likely require re-enumeration of the PCIe buses, and hence, you'd want to make sure all the software was in place on the host to support that.  There is no support for connecting Quartus to multiple boards, other than to use multiple byte-blasters. In my systems, a control processor (PowerPC) and system controller (FPGA) are always powered, and the data processing FPGAs are powered on/off and reconfigured dynamically under software control, eg.,  http://www.ovro.caltech.edu/~dwh/carma_board/ (http://www.ovro.caltech.edu/%7edwh/carma_board/)  Cheers, Dave

[Altera_Forum]

Thanks guys, much appreciated! 

   --- Quote Start ---   ... they can comunicate to each other over the PCIe bus. Since the DE4 has a 4x PCIe 2.0 interface, you can get 4x 5 Gbit/s full duplex for each DE4, assuming the motherboard supports it...   --- Quote End ---   Great. It seems PCIe is the way to go.       --- Quote Start ---   ...What are your data flow requirements? Where is the data coming from? Are you talking MB/s, GB/s, TB/s? Or is the task 'processing' dominated?...   --- Quote End ---   That is a very good question. I forgot to specify that. :oops:   The data will be collected from the network ports on each card ( approx 50% of 1GBps capacity in use per port).    At least 2 Network-ports on each preprocessing-card will be in use. So up to 2x0.5GBps=1GBps of data. The preprocessing of data ( on the inputcards ) could probably double that number.   Thus, worstcase we're talking 2GBps of traffic per card to the main processing card. Numbers might change, but we're talking GBps speeds for sure.       --- Quote Start ---   Do your FPGAs need to be reconfigured dynamically, or can they be configured once at power-on?   --- Quote End ---   Cards does not need to be reconfigured dynamically. Solution will be uploaded to the cards at power-on.       --- Quote Start ---   There is no support for connecting Quartus to multiple boards, other than to use multiple byte-blasters.    --- Quote End ---   So basically the communication has to be handled on a per-boardlevel basis. Fair enough.    I suppose the trick is to delve deeper into documentation of PCIe communication with the Altera board(s).    Any pointers towards good places to start? The altera documentation obviously, but if there are any other/specific sources of information you would reccomend, I'd appreciate the gesture.       Best Regards A.

[Altera_Forum]

Hi, 

 well, obviously, the PCIe Compiler documentation is one place to start. http://www.altera.com/products/ip/iup/pci-express/m-alt-pcie8.html  Some generic information about PCIe, assuming you're not very familiar with it.  PCIe, basically, behaves as a good old PCI: a master/slave memory mapped interface where devices card be either masters, slaves or, most often, both.  Each slave device will present itself as a "memory like" range of addresses. This range will be whatever you wish. You can even expose the in-FPGA and on-card RAM through this range. Masters will request read/writes from a given address range.  So, you'll need to implement suitable PCIe designs in your cards. One solution is to have your data gather cards be just slaves and make your main card be the master. But you can implement whatever suits you best.  Now, before the master card can address the slaves, it needs to be told their address. You'll need to write a small software driver for that. I suggest you start by using Jungo's WinDriver in trial mode. http://www.jungo.com/st/windriver_usb_pci_driver_development_software.html  You may (or not) also want to use the driver to help manage the flow of data from the slaves to the master.

[Altera_Forum]

--- Quote Start ---  ... worstcase we're talking 2GBps of traffic per card to the main processing card. Numbers might change, but we're talking GBps speeds for sure.    --- Quote End ---      --- Quote Start ---    No board-to-board communications? Just board-to-host?  Cards does not need to be reconfigured dynamically. Solution will be uploaded to the cards at power-on.    --- Quote End ---    Then the DE4 will work fine for that.      --- Quote Start ---   Any pointers towards good places to start? The altera documentation obviously, but if there are any other/specific sources of information you would reccomend, I'd appreciate the gesture.    --- Quote End ---      What OS are you planning on using on the host and on the boards (if any)?  My current generation of hardware is housed in compactPCI crates. Each board has a PowerPC interface running Linux. Each crate has an x86 host CPU. The host CPU runs a virtual network driver that turns the PCI connection between host-and-peripheral boards into a network connection. The system is 120 boards housed in 8 crates. The main data flow is control system commands down to all the boards, and data coming back up. There is no need for board-to-board communication, but the network does allow it. This scheme requires an OS at either end, since you are pushing/putting network packets into the OS network stack.  On my older generation of boards. There is a DSP for local control, and its not powerful enough to implement a network stack. In that case, the host CPU programs the DMA controller on each board (a PLX PCI9054 PCI bridge) to move data between the boards and host memory. You could conceivably do something similar if you were not going to have much intelligence on your processing board, i.e., no network stack.  I have not tried accessing my Stratix IV GX kits yet over PCIe. I had expected to be able to play with the default sysfs nodes that get created for each PCI device. These nodes allow you to perform read/write accesses to PCI memory spaces. I'd recommend starting with that method, and blinking an LED :)  Cheers, Dave