PCIe BAR size limitations in Qsys

[Altera_Forum]

Hi, 

 Are there any limitations on the size of each PCIe BAR in Qsys? Or the total size of all BARs?  I have a design using the PCIe compiler in Qsys with 2 BARs on a Stratix IV dev. kit. My aim was to have BAR0:1 address 4 GB of memory and BAR2 access the control/status registers (2 MB).  Under Quartus 11.1sp2/Qsys, I cannot set the BAR0:1 size greater than 1 GB - I get an error to the effect that the "bus translator" address spaces were overlapping. However, with the 1 GB BAR, the design worked fine.  Under Quartus 12.0sp1/Qsys, I can set BAR0:1 to be 4 GB and BAR2 to be 2 MB. There are no warnings or errors from Qsys. However the design doesn't work and causes the PC to hang or produce inconsistent results when I access either BAR space. (I use the Linux sysfs to access the two BARs.) If I change BAR0:1 size to 1 GB, everything works fine. (A BAR0:1 size of 2 GB fails the same way as 4 GB.)  It almost appears as if the two Avalon-MM interfaces corresponding to the two BARs map into the same 32-bit address space and clobber each other. Has anyone run into this situation? I have looked through the PCIe compiler users guide, but didn't find anything.  Thanks, Chandan

[Altera_Forum]

--- Quote Start ---   I have a design using the PCIe compiler in Qsys with 2 BARs on a Stratix IV dev. kit. My aim was to have BAR0:1 address 4 GB of memory and BAR2 access the control/status registers (2 MB).   --- Quote End ---    Have you tried booting your PC with this type of configuration?  I'd say there's a 99% chance your PC will not even boot.  There is very rarely any need to have PCIe (or PCI) BARs bigger than a few kB or MB. If you want to transfer data between 4GB on the board and your host memory, then you would use DMA from the board, in which case the board is the bus master, and the BAR size is of no consequence (since that is used only by the host).  Cheers, Dave

[Altera_Forum]

Thanks for replying. 

   --- Quote Start ---  Have you tried booting your PC with this type of configuration?  --- Quote End ---    Yes. One of our Xeon boxes has a BIOS that lets your place your PCIe resources above the 4GB mark and allows up to a total of 1 TB of PCIe space. "lspci" under Linux even shows that the BAR size of the dev. board is 4 GB as does the sysfs entry.  The really frustrating thing is that, occasionally, with some builds I can actually get it to work and access all 4 GB of the BAR. (I can tell because the design has instrumentation that I can get to through BAR2.). My first thought was a timing problem, but looking carefully through the timing report doesn't show anything.    --- Quote Start ---  There is very rarely any need to have PCIe (or PCI) BARs bigger than a few kB or MB.  --- Quote End ---    Yes, I can probably use DMA to bypass the BAR mechanism completely. We are building a memory subsystem and this dev. board is being used more like an accelerator for simulations. I would prefer not to add extra logic and software between the design and the testbench. Using mmap() to make it appear like a chunk of memory seems more natural.  Thanks, Chandan

[Altera_Forum]

--- Quote Start ---   Yes, I can probably use DMA to bypass the BAR mechanism completely. We are building a memory subsystem and this dev. board is being used more like an accelerator for simulations. I would prefer not to add extra logic and software between the design and the testbench. Using mmap() to make it appear like a chunk of memory seems more natural.   --- Quote End ---  You need to use a DMA controller to generate burst transactions over the PCIe bus. You cannot generate burst transactions over PCI/PCIe (except maybe some simple ones by generating 128-byte transactions or playing with cache-lines).  Your two options for a DMA controller are; the host (root-complex) or the device (end-point). Unless your Xeon has a DMA controller, you need to use a DMA controller on the PCIe board.  Cheers, Dave