pci express sopc vs qsys

[Altera_Forum]

Hey !! 

 I´ve made a fpga design in sopc which has got an pci express hard_ip block and several I/O´s. My plan is to serve those I/O´s from a host pc, which contains a linux operating system. my bar0 (connected to my I/O ports) base address is given by 0x00100000 (auto assigned by quartus). my I/O (it´s an avalon slave) has got base address of 0x00180000.  I was able to read/write from/to the I/O port by executing a read/write operation to the offset address of 0x80000 (difference of avalon slave address and bar base address) on my host pc.  Now i changed to QSYS. With the same settings compared to SOPC system i get an base address for bar1 of 0x00200000 (auto assigned by QSYS), base address of bar0 is 0x0000, base address of avalon interface is unchanged. Anyway two things are confusing me: The base address and size of bar1 is changing when interfaces, which are connected to bar0, change, in my example hsmc_bank2_west ???? Why is the bar base address higher than the avalon base address ???  To which offset address do i have to perform read/write operations of I/Os on my host pc ???  Does anybody have an idea ?? thanks for your help

[Altera_Forum]

Personally I wouldn't use auto-assigned addresses for anything other than trivial designs. 

 If you assign addresses yourself you can group them sensibly - which may be necessary if you find you need to add a bridge for any reason.  Your host always has to do accesses relative to the base of the BAR (ie add the offset to the address returned by ioremap_uncached()). On the FPGA the PCIe master interface uses the address bits (only the width of the BAR) to select between its avalon slaves.  So the size of the BAR (always a power of 2) is all that matters. In some sense the PCIe's Avalon master interface supplies the high address bits, but since the Avalon 'bus' is slave arbited they are never actually used!  Although the SOPC builder and QSYS give each slave an address, the address is a property of the master interface. There is nothing technically stopping different masters using different addresses for the same slave! (It would be confusing though - which may be why it isn't allowed).

[Altera_Forum]

hey !! 

thanks for ur reply. anyway i can´t change the base adress of my bar, this one is automatically set by quartus or qsys, i haven´t found a way to set manually so far. but what´s my offset address if the bar base adress is higher than the address of my avalon slave ?? if i change the avalon slave base address the bar base address changes as well !!  thanks

[Altera_Forum]

--- Quote Start ---   Although the SOPC builder and QSYS give each slave an address, the address is a property of the master interface. There is nothing technically stopping different masters using different addresses for the same slave! (It would be confusing though - which may be why it isn't allowed).  --- Quote End ---    Actually, it is allowed, its just not very obvious in the Qsys GUI.   Each master can have a different address map. The address map you see in the 'System Contents' page of the GUI is the 'default' address map where all masters decode the slaves with same address map. If you click on the 'Address Map' tab, then you can edit the addresses of the slaves on a per-master basis. When you do this, the addresses of slaves with different master addresses changes on the 'System Contents' page to 'mixed'.  If you want to minimize the BAR decode size, you need to move the slave addresses accessed via a specific BAR to lowest possible addresses.  In my opinion, this is a poor feature of the Qsys PCIe bridge. Most PCIe bridges have inbound translation windows, which allow you to move a BAR over any address in the local address map. The bridges have a (small) BAR for control registers (lets call it BAR0), and then other user-programmable BARs (eg., BAR1). The BAR1 size can be configured at power-on, and then the BAR can be moved by setting the base address in the control registers in BAR0.  Cheers, Dave

[Altera_Forum]

We have 3 BARs (0, 2, 3 and 4) - largely to do with no one standing back from the design far enough! since BAR4 covers 16MB SDRAM and some IO and everything else would fit in the spare space. 

The other BARs are much smaller.  One of the BARs maps some stuff that has avalon addresses between 0x40000000-0x4003ffff this is (probably) a 256b BAR.  There was a problem with BAR4, the signals are fed through a conduit because the auto-assigned BAR ended up having extra unwanted address lines and was larger than 32MB (I'm not sure of the exact details).  Actually, you could probably use a conduit to feed in the high address lines between the PCIe Avalon masters and any avalon slaves.  We do currently has an 'interesting' issue when we do PCIe reads of locations that don't have an Avalon slave. Not sure what the fpga does, but it sure confuses the little ppc at the other end!

[Altera_Forum]

thanks all.. 

anyway but i can´t change, for example, my 'avalon base address start' of my bar0 ?! as i wrote in the first post, i think this value is necessary for writing to i/o´s connected to this bar from my host system (for my offsetadress of the i/o´s in linux), i think i need to know the translation table, how adresess are mapped from outside the pci express block to address connected to each bar. why is this made in qsys so much different to sopc, in sopc everything works fine :-(  greets

[Altera_Forum]

--- Quote Start ---   anyway but i can´t change, for example, my 'avalon base address start' of my bar0 ?!   --- Quote End ---  BAR0 is a master, it has no Avalon base address. Only slaves have addresses, and those slaves can be addressed on a per-master basis as I describe above.  For example, lets say you think of your system as having only a single address map, where the address map is that seen by some local master, like a NIOS processor, eg., the address map seen by the NIOS processor (Avalon master) is;  0x0000 RAM 0x1000 Registers 0x2000 RAM  Now lets say you want to access only the registers via BAR0. If the registers are the only slave you map to BAR0, then by default, they will appear at an offset of 0x1000 into BAR0. However, using the 'Address Map' tab, you can set the base address of the registers to 0x0000 for the BAR0 master.  With this setup, the NIOS processor will see the registers at 0x1000 in its address map, and the BAR0 master will see the registers at 0x0000.  Cheers, Dave

[Altera_Forum]

hey dave.. 

 thanks, but i think that the bar has got an avalon address, as u can see in the picture of my first post. the message section of qsys says: bar0, avalon base address start: .....  i think i know how addresses are mapped let´s say inside the fpga between avalon masters and avalon slaves. but my idea was that there must be also an address mapping between the pci express and my os on the host pc. when i want to serve i/o s from that os i don´t know which base address (or offset address) i have to write to from outside the fpga to reach the appropriate i/o. :confused: actually it was quite obvious in sopc....  cheers,

[Altera_Forum]

--- Quote Start ---   there must be also an address mapping between the pci express and my os on the host pc. when i want to serve i/o s from that os i don´t know which base address (or offset address) i have to write to from outside the fpga to reach the appropriate i/o. :confused:   --- Quote End ---    That is the purpose of the BAR registers in the PCI/PCIe configuration space. When your computer boots, the BIOS writes the base address of the BAR as seen by the host CPU, eg., an x86 CPU.  If you are using Linux, you can list these addresses using lspci. If you are using Windows, you can use PCITree. Linux has a default device driver you can use to perform read/write accesses to BAR registers. I don't write Windows drivers, so you are on your own there. I guess most people use the Jungo driver support.  If you want to see an example of the output of lspci for a Qsys PCIe bridge, look at the document I link to in this thread:  http://www.alteraforum.com/forum/showthread.php?t=35678  In the zip file under the code/ folder is a program called pci_debug that lets you access the PCIe BARs under Linux.  Cheers, Dave

[Altera_Forum]

thanks Dave for ur help, with ur explanations i did it :-). pci express works now in qsys, heading forward to dma transfers now, hope u can help me with that when i´ve got problems ;-) 

 great job, thanks again :-)  greets

[Altera_Forum]

--- Quote Start ---  thanks Dave for ur help, with ur explanations i did it :-). pci express works now in qsys,   --- Quote End ---  Great!    --- Quote Start ---    heading forward to dma transfers now, hope u can help me with that when i´ve got problems ;-)   --- Quote End ---  Depending on how you implement the DMA controller you will face a few issues.  Lets say you ask for a 4kB page in the host memory. That memory can be mapped from its virtual address to the physical address. Lets say its address is the 64-bit value 1234_5678_0000_0000h. Now how do you DMA from your Qsys system into that page?  The Qsys PCIe example implements a DMA controller within the Qsys system, and the Qsys system has only 32-bit addresses. Since the DMA controller can only deal with 32-bit Avalon addresses, how then can you access the 64-bit address of the host?   The CRA registers should have a register that allows you to program the most-significant address bits of the TXS slave address. For example, if you setup a single TXS 1MB window, then the 10-LSBs of the address would come from the Avalon TXS access address, while the 64-10 = 54 MSBs should come from a register in the CRA slave.   I haven't used this feature during my testing, but I'm pretty sure that should be how it works.  Cheers, Dave