Help: Windows didn't assign Bar0 to PCie (address mapping is not correct)

[Altera_Forum]

Dear all, 

 We changed the Altera Cyclone PCIe DDR2 reference design to our needs. However, it doesn't work. The address mapping is not correct, it doesn't agree to address assignments in Qsys.  In WinDriver, Bar0 doesn't appear, only Bar2 appears under "Memory" tab (see Attached Figure). But the address range of Bar2 is not correct either, and the reading and writing were not correct when we write then read from Bar2.  Anyone can give any suggestions? Why it didn't have bar0 as assigned in Qsys?  The Windows operating system is Windows XP.  I attached the figures of Qsys block diagram, Qsys address map and IP Complier.  Thank you very much!

[Altera_Forum]

Look at your "Address Map" image. BAR0 has the 256MB DDR mapped from 0x1000_0000 to 0x1FFF_FFFF, and it has the onchip RAM mapped lower in that are, so the BAR region needs to be 2 x 256MB = 512MB. 

 Many computers will not boot with such large BAR regions. In your case, the BIOS probably did not enable that region.  Start by trying to use a smaller BAR.  The Qsys PCIe bridge is missing a function that most PCI bridges normally have, which is the ability to independently set the BAR size, and the base address of where that BAR maps to. Using this feature, you can normally take a 1MB BAR, and move it anywhere in the PCIe device address map, eg., in your case you would be able to look at the DDR 1MB at a time. This is sufficient for simple testing. Since the final application should use DMA, the use of small BAR regions is not an issue.  Start with a small BAR - even if you think this is not the right solution - since using the small BAR will show you whether or not this is really your problem.  Cheers, Dave

[Altera_Forum]

Thanks, can the bar region be as large as 256 MB or more? 

 We use "system"-> "assign base address" menu, the DDR now is mapped to 0x0000_0000 to 0x0fff_ffff, the on-chip memory now is mapped to 0x1000_0000 to 0x1003_ffff.   But it still doesn't work. Bar0 still does not appear.      --- Quote Start ---  Look at your "Address Map" image. BAR0 has the 256MB DDR mapped from 0x1000_0000 to 0x1FFF_FFFF, and it has the onchip RAM mapped lower in that are, so the BAR region needs to be 2 x 256MB = 512MB.  Many computers will not boot with such large BAR regions. In your case, the BIOS probably did not enable that region.  Start by trying to use a smaller BAR.  The Qsys PCIe bridge is missing a function that most PCI bridges normally have, which is the ability to independently set the BAR size, and the base address of where that BAR maps to. Using this feature, you can normally take a 1MB BAR, and move it anywhere in the PCIe device address map, eg., in your case you would be able to look at the DDR 1MB at a time. This is sufficient for simple testing. Since the final application should use DMA, the use of small BAR regions is not an issue.  Start with a small BAR - even if you think this is not the right solution - since using the small BAR will show you whether or not this is really your problem.  Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---  can the bar region be as large as 256 MB or more?   --- Quote End ---    The PCIe spec will not stop you, but your particular hardware might. My EliteBook will not boot with a BAR of 256MB.    --- Quote Start ---   We use "system"-> "assign base address" menu, the DDR now is mapped to 0x0000_0000 to 0x0fff_ffff, the on-chip memory now is mapped to 0x1000_0000 to 0x1003_ffff.  But it still doesn't work. Bar0 still does not appear.  --- Quote End ---    Because you did not change the BAR size by doing this. Look at the report on the Qsys GUI. Look at the synthesis report.   Look at this document here:  http://www.alteraforum.com/forum/showthread.php?t=35678  It has some comments.  To try and isolate this problem, start with just the on-chip RAM in the BAR at a base address of zero. You can use the "Address Map" tab to re-assign the addresses.   Once you can confirm that you can boot windows, you can increase the size of the BAR (the offset of the on-chip RAM) until it breaks. Then you will know the limits for your particular hardware setup.  Cheers, Dave

[Altera_Forum]

Thank you, Dave, 

 We actually got rid of the DDR2 once and has on-chip memory and SGDMA at Qsys. But it didn't seem to help and Bar0 sill didn't appear at the WinDriver memory tab. There was nothing under WinDriver memory tab.  Could you diagnose this problem?     --- Quote Start ---  The PCIe spec will not stop you, but your particular hardware might. My EliteBook will not boot with a BAR of 256MB.    Because you did not change the BAR size by doing this. Look at the report on the Qsys GUI. Look at the synthesis report.   Look at this document here:  http://www.alteraforum.com/forum/showthread.php?t=35678  It has some comments.  To try and isolate this problem, start with just the on-chip RAM in the BAR at a base address of zero. You can use the "Address Map" tab to re-assign the addresses.   Once you can confirm that you can boot windows, you can increase the size of the BAR (the offset of the on-chip RAM) until it breaks. Then you will know the limits for your particular hardware setup.  Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---   Could you diagnose this problem?  --- Quote End ---    Go through the tutorial I wrote in the thread I pointed you to.  The examples in the document are for x8 and x1, and there is a x4 PCIe in the example code. The boards are the Stratix IV GX Development kit and the Cyclone IV GX Transceiver Starter kit. There's enough detail in there for you to create a design for your specific board.  Under code/pci_debug is a tool for accessing the BARs under Linux. You can use it to read/write to the on-chip RAM, and then later, when you add a DDR controller to the design, use it for accessing DDR.  Cheers, Dave

[Altera_Forum]

--- Quote Start ---   The Windows operating system is Windows XP.   --- Quote End ---    Could this be a 32-bit vs 64-bit issue? What is the processor you are using?  Perhaps try setting BAR0 as 32-bit to see if it makes a difference.  To test Linux, download a bootable CD-ROM and boot from that, or install Ubuntu via Wubi (which sticks a file onto your windows drive).  Cheers, Dave

[Altera_Forum]

--- Quote Start ---   Bar0 sill didn't appear at the WinDriver memory tab. There was nothing under WinDriver memory tab.   --- Quote End ---  Ignore WinDriver to start with. Use PCITree to see if the BIOS programmed the 64-bit BAR0 correctly.  Who knows if your copy of WinDriver is setup to allocation 64-bit BARs? Read the documentation and see.  Cheers, Dave

[Altera_Forum]

Dave, thank you for so many valuable suggestions. 

   --- Quote Start ---  Ignore WinDriver to start with. Use PCITree to see if the BIOS programmed the 64-bit BAR0 correctly.  Who knows if your copy of WinDriver is setup to allocation 64-bit BARs? Read the documentation and see.  Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---  thank you for so many valuable suggestions.  --- Quote End ---    Hopefully one of them gives you some insight as to what is happening.  Cheers, Dave

[Altera_Forum]

Thanks, Dave.  

 Then how to use Qsys to connect to large memory chip (DDR II)?   If I assign large Bar size covering the whole address of DDR II, computer will not boot.   What to do to make large memory chip to work with PCIe?  Thank you very much!     --- Quote Start ---  The PCIe spec will not stop you, but your particular hardware might. My EliteBook will not boot with a BAR of 256MB.    Because you did not change the BAR size by doing this. Look at the report on the Qsys GUI. Look at the synthesis report.   Look at this document here:  http://www.alteraforum.com/forum/showthread.php?t=35678  It has some comments.  To try and isolate this problem, start with just the on-chip RAM in the BAR at a base address of zero. You can use the "Address Map" tab to re-assign the addresses.   Once you can confirm that you can boot windows, you can increase the size of the BAR (the offset of the on-chip RAM) until it breaks. Then you will know the limits for your particular hardware setup.  Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---   Then how to use Qsys to connect to large memory chip (DDR II)?   If I assign large Bar size covering the whole address of DDR II, computer will not boot.   What to do to make large memory chip to work with PCIe?   --- Quote End ---    You need to use slight-of-hand and indirection :)  You need to think of the memory map of your board as being independent/separate to the memory map of your x86 host. If you want to move results between your Qsys memory map and your x86 host memory, then you use DMA. The DMA controller needs to be able to see both PCIe addresses and Qsys addresses to be efficient.  If you use the x86 host to directly read and write to your Qsys system, it will not be very efficient (low data rate). Typically you use (slow) x86 transactions to setup the DMA controller, and then let it perform efficient transfers.  Cheers, Dave

[Altera_Forum]

but will it reduce BAR0 size? I just concern if my computer can boot.  

Do you mean using DMA can reduce BAR0 size if the x86 host memory is still connected to Qsys ?      --- Quote Start ---  You need to use slight-of-hand and indirection :)  You need to think of the memory map of your board as being independent/separate to the memory map of your x86 host. If you want to move results between your Qsys memory map and your x86 host memory, then you use DMA. The DMA controller needs to be able to see both PCIe addresses and Qsys addresses to be efficient.  If you use the x86 host to directly read and write to your Qsys system, it will not be very efficient (low data rate). Typically you use (slow) x86 transactions to setup the DMA controller, and then let it perform efficient transfers.  Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---  but will it reduce BAR0 size? I just concern if my computer can boot.  Do you mean using DMA can reduce BAR0 size if the x86 host memory is still connected to Qsys ?  --- Quote End ---    The BAR size will be reduced to whatever the common set of control registers requires.  Consider the following simple case. Lets say I have a DMA controller that has the following registers; 64-bit PCIe address, 32-bit Avalon (Qsys) address, direction (PCIe-to-Avalon or Avalon-to-PCIe), length in bytes, a control register and a status register. You need 7 32-bit registers to describe this. The minimum practical BAR size is about 256-bytes, but its more typical to use a page of the host memory, i.e., 4kB or 8kB.  You can program the DMA controller to move a block of data to or from any PCIe address (which includes the host memory) to an Avalon address, which includes your DDR memory.  Note how the host does not need to see the Avalon DDR memory, it is the DMA controller that needs to see those addresses.  Cheers, Dave

[Altera_Forum]

Thank you! Dave, 

 Do you mean DDR2 is not connected to PCIe Hard IP, it is only connected to DMA? What is the differences between PCIe address and Avalon address? Isn't PCIe address also Avalon address since it often uses Avalon MM interface?   Also, do you have examples/more detailed documents about this?  Thank you very much!     --- Quote Start ---  The BAR size will be reduced to whatever the common set of control registers requires.  Consider the following simple case. Lets say I have a DMA controller that has the following registers; 64-bit PCIe address, 32-bit Avalon (Qsys) address, direction (PCIe-to-Avalon or Avalon-to-PCIe), length in bytes, a control register and a status register. You need 7 32-bit registers to describe this. The minimum practical BAR size is about 256-bytes, but its more typical to use a page of the host memory, i.e., 4kB or 8kB.  You can program the DMA controller to move a block of data to or from any PCIe address (which includes the host memory) to an Avalon address, which includes your DDR memory.  Note how the host does not need to see the Avalon DDR memory, it is the DMA controller that needs to see those addresses.  Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---   Do you mean DDR2 is not connected to PCIe Hard IP, it is only connected to DMA?   --- Quote End ---  DDR2 and DMA and the Qsys PCIe bridge would be connected to a Qsys system. However, its the operation of a PCIe-to-Qsys bridge that you need to understand.    --- Quote Start ---   What is the differences between PCIe address and Avalon address? Isn't PCIe address also Avalon address since it often uses Avalon MM interface?    --- Quote End ---  No, PCIe addresses and Qsys addresses are two completely separate address maps.  Consider the case of 4 identical FPGA boards plugged into a motherboard. Those four boards could be configured with identical Qsys systems, right? Their Qsys address maps are therefore identical. How can the PCIe address map be identical to the Qsys address map? It cannot.  In this case, how would each of the boards DMA into x86 memory? Well, the host would access each of the board's DMA controllers via registers mapped into a BAR. Each board would be uniquely mapped into the x86 address map by the BIOS. The Qsys source address on each board could be identical, but the PCIe destination address would be unique to each board.  Draw a picture of the address maps, and you'll see that a DMA controller is really a 'bridge' that can see the Qsys address map on one side, and the PCIe address map on the other.    --- Quote Start ---   Also, do you have examples/more detailed documents about this?   --- Quote End ---  No. This is pretty standard for PCI development. However, its not exactly documented anywhere, its more like "that is just the way it is". Once you understand it, it makes sense.  Read the data sheet of the PLX PCI9054 PCI bridge, or read the PCI section of the Freescale MPC8349EA PowerQuicc II Pro processor documentation, and you'll see very similar descriptions of the operation of the PCI bridges.  Cheers, Dave

[Altera_Forum]

Thank you! Dave, another question: 

 How to have a "Qsys PCIe bridge"? because I can not find it in Qsys component library. what is the exact name? or do we need to build one ourselves?  Thank you very much!    --- Quote Start ---  DDR2 and DMA and the Qsys PCIe bridge would be connected to a Qsys system. However, its the operation of a PCIe-to-Qsys bridge that you need to understand.   Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---   How to have a "Qsys PCIe bridge"? because I can not find it in Qsys component library. what is the exact name? or do we need to build one ourselves?   --- Quote End ---  I'm investigating the Altera IP at the moment:  http://www.alteraforum.com/forum/showthread.php?t=35678  The existing Qsys component does not implement a DMA controller as part of the bridge, so it does not meet the requirements for what I would consider to be a decent bridge. I need to complete the MegaWizard flow review and then some of the SOPC components on the Altera wiki. If none of them have features that meet my requirements, I'll create a design. I'll then document it and post it to the Altera wiki. I'm currently working on a board design, so might not get started on the bridge design for a little while (since it does not impact the board layout, just the resources used in the FPGA).  What exactly are the requirements of your design? Perhaps I or someone else on the forum can suggest a solution that will get you working now.  Cheers, Dave

[Altera_Forum]

Thank you, Dave. 

 We have a PCIe Hard IP, 128M 16 bit DDR2 SDRAM and 21K 64-bit on-chip memory.   Qsys assigns 256MB for the DDR2 SDRAM due to Avalon MM interface, plus on-chip memory, the total size of Bar0 exceeds 256 MB, and we can not make it work with WinDriver and our computer (Windows XP).   It seems the maximum size of Bar0 for our computer is 256 MB.   So we want to make both DDR2 SDRAM and on-chip memory working, we want to reduce the size of Bar0.      --- Quote Start ---  I'm investigating the Altera IP at the moment:  http://www.alteraforum.com/forum/showthread.php?t=35678  The existing Qsys component does not implement a DMA controller as part of the bridge, so it does not meet the requirements for what I would consider to be a decent bridge. I need to complete the MegaWizard flow review and then some of the SOPC components on the Altera wiki. If none of them have features that meet my requirements, I'll create a design. I'll then document it and post it to the Altera wiki. I'm currently working on a board design, so might not get started on the bridge design for a little while (since it does not impact the board layout, just the resources used in the FPGA).  What exactly are the requirements of your design? Perhaps I or someone else on the forum can suggest a solution that will get you working now.  Cheers, Dave  --- Quote End ---  

[Altera_Forum]

--- Quote Start ---   We have a PCIe Hard IP, 128M 16 bit DDR2 SDRAM and 21K 64-bit on-chip memory.   Qsys assigns 256MB for the DDR2 SDRAM due to Avalon MM interface, plus on-chip memory, the total size of Bar0 exceeds 256 MB, and we can not make it work with WinDriver and our computer (Windows XP).   It seems the maximum size of Bar0 for our computer is 256 MB.   So we want to make both DDR2 SDRAM and on-chip memory working, we want to reduce the size of Bar0.  --- Quote End ---    That tells me what the hardware is, but it does not tell me anything about why. Here's a few questions;  1) What FPGA do you have on the board?  2) Why does the FPGA have memory? What is it to be used for?  3) Why does the host CPU need to see that memory?   4) What is your data flow?  For example, in my application I have high-speed ADCs and signal processing logic in the FPGA. The FPGA results will be written to DDR3 RAM, and then those results DMAed to the host CPU. There is a few MB of data that needs to be transferred every ~20ms. However, those results can be queued, and transferred as a larger DMA transfer every 500ms.  The host CPU will run Linux, and a device driver will setup the DMA controllers on multiple boards to DMA their results into main memory. The main memory will contain scatter-gather DMA buffers. The data will be moved out of the buffers once the host has processed the results.  You need to come up with a similar data flow description for your system.  Cheers, Dave

[Altera_Forum]

1) It is Cyclone IV GX speed grade 7, I modified the design in  

http://www.alterawiki.com/wiki/pci_express_in_qsys_example_designs changed the DDR2 to our 128 M 16 bit DDR2 SDRAM.  2) The memory is to be used for storing some parameters/commands for signal processing  3) There is a lot of RF data from ADCs for the FPGA to process, CPU needs to send commands/parameters, also obtains processed data storage in DDR2 SDRAM from PCIe. I don't know if CPU needs to see that memory or not, if it does not need to see that memory, how to realize ? to get the data from DDR2 SDRAM?  4) FPGA processes the ADC data,CPU sends commands/parameters, access data in on-chip memory or in DDR2 SDRAM. displays processed data for real-time (30frams/s) images on screen.     --- Quote Start ---  That tells me what the hardware is, but it does not tell me anything about why. Here's a few questions;  1) What FPGA do you have on the board?  2) Why does the FPGA have memory? What is it to be used for?  3) Why does the host CPU need to see that memory?   4) What is your data flow?  For example, in my application I have high-speed ADCs and signal processing logic in the FPGA. The FPGA results will be written to DDR3 RAM, and then those results DMAed to the host CPU. There is a few MB of data that needs to be transferred every ~20ms. However, those results can be queued, and transferred as a larger DMA transfer every 500ms.  The host CPU will run Linux, and a device driver will setup the DMA controllers on multiple boards to DMA their results into main memory. The main memory will contain scatter-gather DMA buffers. The data will be moved out of the buffers once the host has processed the results.  You need to come up with a similar data flow description for your system.  Cheers, Dave  --- Quote End ---