FPGA configuration time and PCIe secification

[Altera_Forum]

Has anyone used the PCIe interface of a Cyclone device while configuring it using the AS (active serial) scheme? 

 I'm using a Cyclone IV device (EP4CGX30F484) and planning to use its PCIe interface. I've read from the PCIe specification that the minimum time between the PC's power rails being stable and the PCIe PERST# signal being deactivated is T_PVPERL = 100 ms (acitvating and deactivating PERST# causes a reset of the PCIe interface and causes PCIe lanes to initialise). You get another 100 ms from receiving the PERST# signal before the sequence starts, so that's a total of 200 ms. The PCIe specification only states a minimum time between power being stable and sending the PERST# signal, so you could get a few seconds before you have to be ready, but I don't know if I can rely on this.  Using the active serial configuration scheme would take me 1.225 s, so it seems I have to use a different scheme, such as FPP. Altera recommend this in their application note AN529:  “For Cyclone IV GX devices to meet the PCIe 100 ms wake-up time requirement, you must use PS configuration mode for the EP4CGX15, EP4CGX22, and EP4CGX30 (except for F484 package) devices and FPP configuration mode for the EP4CGX30 (only for F484 package), EP4CGX50, EP4CGX75, EP4CGX110, and EP4CGX150 devices.”  FPP is a pain because you need another chip to manage the configuration process and this chip (MAX II or a microprocessor) needs its own software.  So my question is: has anyone got the PCIe interface to work reliably while their configuration takes over 200 ms? I'd also like to hear if you haven't! Thanks.

[Altera_Forum]

The Answer - I havent used a cyclone for it, but we did use a stratix 2 GX in this scheme. The result - some boards that wont boot in some motherboards - and it wasnt even a specific manufacturer thing - some boards worked and some didnt. 

 The larger devices take a long time to program, so will fall outside the PCIe spec. You HAVE to use FPP or you're going to have problems. But Max IIs can do it and are very cheap and very small. The PCIe dev boards all use the FPP scheme if you need a reference design. Theres even a free FPP block you can pull out of the megawizard, so design is pretty small.

[Altera_Forum]

Thanks for your answer Tricky. Do you know how big your configuration file was, or how long it took to load your configuration via AS?

[Altera_Forum]

Hi 

It wasnt my project but I know there were problems. IIRC, the config time was fairly close to the spec, 200ms, hence it working most of the time.   If you're not up in time, a PC or similar is never going to see the board.

[Altera_Forum]

--- Quote Start ---  Has anyone used the PCIe interface of a Cyclone device while configuring it using the AS (active serial) scheme?  I'm using a Cyclone IV device (EP4CGX30F484) and planning to use its PCIe interface. I've read from the PCIe specification that the minimum time between the PC's power rails being stable and the PCIe PERST# signal being deactivated is T_PVPERL = 100 ms (acitvating and deactivating PERST# causes a reset of the PCIe interface and causes PCIe lanes to initialise). You get another 100 ms from receiving the PERST# signal before the sequence starts, so that's a total of 200 ms. The PCIe specification only states a minimum time between power being stable and sending the PERST# signal, so you could get a few seconds before you have to be ready, but I don't know if I can rely on this.   --- Quote End ---  You actually need to read several specifications to track down the timing parameters. I've attached a few pages from a document that cross-references the information you're interested in.  Unfortunately the answer to the question "How soon does the PCIe end-point need to be ready?" doesn't really have a simple answer, since it depends on your application. In the CPCI-S.0 application I'm working on, the PCIe end-point just has to be ready before the host processor deasserts PCIe RST#, and from hardware measurements on the CPUs I intend to use, it appears I have 800ms. This is lucky for me, as the peripheral board needs to use a hot-swap controller, and hot-swap controller all seem to have power-on delays of 100ms!  The power-on timing is harder for PCIe motherboard applications, as there you really have no control over the motherboard reset timing, so you need to be ultra-conservative and meet the TPVPERL = 100ms specification.  Cheers, Dave

[Altera_Forum]

--- Quote Start ---   1). So, how much time is available for configuration: 100 ms or 200ms?   --- Quote End ---   The design target for a product should be under 100ms. If you are designing the board for your own system, and you have a specific host CPU or motherboard you are planning to use, then measure the PCI RST# time on the motherboard and design based on that.    --- Quote Start ---   2). How to calculate the total configuration time?  POR can be taken from CIV datasheet and it is 3ms for Fast POR  What other time should be taken into account to calculate the total configuration time and where in CIV datasheet is it specified?  --- Quote End ---    The total configuration timing sequence is;  1) External power gets turned on to the "system"; motherboard or chassis, main host CPU, and PCIe bridges and devices.  2) The power supplies for the host and the peripheral devices start ramping up.   The timing for this part is completely arbitrary and depends on the system design.   In the case of your FPGA-based PCIe end-point, you'll have whatever ramp time your power supplies require, and if you've specified a fast POR, they need to ramp within that time.  3) Once the power supplies are in spec, the FPGA configures, eg., via Active Serial, Passive Serial, or Fast Passive Parallel.  4) The BIOS or host CPU bootloader enumerates the PCIe bus.  So long as (3) completes before (4), your system will work, otherwise it will not, as the PCIe device will not be enumerated.  Cheers, Dave

[Altera_Forum]

thought i'd throw out another alternative: Cyclone V and autonomous HIP. this configures the FPGA via AS in 2 stages, the periphery and the core. loading the periphery first allows you to meet PCIe specs, then the FPGA goes out and reads the core image: 

 http://www.altera.com/literature/ug/ug_cvp.pdf

[Altera_Forum]

I thought about it regarding Cyclone IV - but apparently in Cyclone IV this is not possible...

[Altera_Forum]

CvP/autonomous HIP is a 28nm/V family feature

[Altera_Forum]

The 'split config' doesn't work in any available Cyclone IV devices. 

IIRC It is supposed to be functional in the rev C part - not yet available.

[Altera_Forum]

--- Quote Start ---  The 'split config' doesn't work in any available Cyclone IV devices. IIRC It is supposed to be functional in the rev C part - not yet available.  --- Quote End ---   In what Altera document can I find more information about it?

[Altera_Forum]

Can anyone help with this related question ... I have a DK-DEV-5SGXEA7N which is the Stratix V FPGA development kit running the example PCIe designs hip_s5gx_x1_g2_ast64_5SGXEA7K2F40_121SP1.qar 

The host system is a HP Z220 deskside system with various PCIe slots. To date , I have not been successful in getting the development card recognized or the altpci_demo_63 application to see the card. I have tried various lane sized and stuck to Gen2 speed. I need a X1 -> X16 riser card to be able to try the X1 slots.  I believe the card worked fine in another PCIe system and am wondering if there is some race between the training and the BIOS comming up ? I have tried to stall the BIOS by interrupting it , but would,need to probe further to see if the Intel chipset is trying to train while the FPGA is still configuring.  Once Windows 7 is up I have run several utilities that probe the PCIe slots, and I believe the utility says the slot is seen as having something in it by the present pin, but says the bus02 is empty. Asking the utility, PCIscope, to read the slot comes up with all F's for all registers and sometines an error "Master Abort".  One other thing, the 4 lane LEDS on the top of the card user_LED[4..7], I have seen all come on then, when I continued BIOS, one led went out. Also user LED[0] someties appears to have both gren and red on together .  Any ideas as this is a loaner card and I can't make much progress if the application program can't see the card .  I was able to modify the alive_led flash rate indicating the system could re-build and get the FLASH re-programmed.

[Altera_Forum]

--- Quote Start ---    am wondering if there is some race between the training and the BIOS comming up ?   --- Quote End ---    If you were able to build the design, then try adding a SignalTap II instance and probing PCI RST#.   Perform a warm reboot of your system, and you should see a rising-edge on RST#. That indicates your FPGA was alive before the BIOS attempted to enumerate, so the issue is not a race condition.  If you can get the card to boot ok on warm reboot, but not at power-on, then you might have a race condition.  Cheers, Dave

[Altera_Forum]

Thanks dwh ... 

 I was able to ge the Signaltrap into the design and have it trigger. The ltssm seems to be cycling through states 0, 1 and 2. I still need to add the RST# signal ... included and the Windows 7 seems to take the power supplies down on a restart ... I have tried to check the POST / BIOS for a configuration to disable this but have not found one.  I have tried to interrupt the POST / BIOS but it didn't seem to make any difference ...   I believe I need to configure the Signaltrace to FLASH to see if I can capture the rising edge of RST#  Otherwise I will need to check the rising edge of RST# and the signal indicating the FPGA is configured and running.

[Altera_Forum]

--- Quote Start ---   I still need to add the RST# signal ... included and the Windows 7 seems to take the power supplies down on a restart ... I have tried to check the POST / BIOS for a configuration to disable this but have not found one.   --- Quote End ---   I haven't seen that in the system I tested with.    --- Quote Start ---   I have tried to interrupt the POST / BIOS but it didn't seem to make any difference ...   I believe I need to configure the Signaltrace to FLASH to see if I can capture the rising edge of RST#  Otherwise I will need to check the rising edge of RST# and the signal indicating the FPGA is configured and running.  --- Quote End ---    SignalTap II has a power-on trigger option. You can configure SignalTap II to trigger on the rising-edge of RST#, and then download the trace at a later time.  I'm pretty sure I included traces captured using this method in the PDF in this thread:  http://www.alteraforum.com/forum/showthread.php?t=35678  Cheers, Dave

[Altera_Forum]

Thanks Dave.  

 I did set the power-on trigger in the SignalTap II instance , committed to FLASH, powered the system on and in the Signaltap session, tried to "Read Data" resulting in a message like trigger not seen.  As a sanity check, I changed the trigger to be the rising edge of the alive_led, recompiled , committed to FLASH and powered the system on. This time in the SignalTap session, the "Read Data " operation returns the trace buffer with the trigger condition at the center indicating the trigger was hit.  If this is a case of the FPGA still configuring when RST# at the slot is deasserted, I'm not even sure delaying POST will help.  1. RST# may be required to initialize logic and it is never seen active when the FPGA logic is running.  As a confirmation, I would like to scope out the power ramp, RST# signal and the FPGA DONE signal or whatever indicated the FPGA is configed and running functional   clocks.  To test this , I figure I may be able to provide a local reset via a user switch so I can reset the FPGA manually. ( OR'ing the slot RST# with my on board reset )    Altera has introducted CvP , in part I believe as the text below indicates ... past issues with power up timing vs FPGA configuration times. "....Altera’s new device configuration mode—configuration via protocol (CvP)—can be used with PCI Express® to configure the core fabric of Altera’s 28-nm Arria® V, Cyclone® V, and Stratix® V FPGAs. CvP can reduce product cost and board size, while simplifying the software usage model, and providing robust in-field system upgrade capability. In addition, the autonomous, embedded PCIe IP core helps ensure that designs meet PCIe power-up time requirements, irrespective of the FPGA core fabric configuration time, guaranteeing a wide range of interoperability with various PCIe-based computer platforms....."    2. Could the Intel PCIe chipset enumerate the PCIe slots only once and then disabled the slot ?  To investigat this I figure I will call HP support to see if they know anything about it.  Dave, any other ideas on your end ?

[Altera_Forum]

--- Quote Start ---   I did set the power-on trigger in the SignalTap II instance , committed to FLASH, powered the system on and in the Signaltap session, tried to "Read Data" resulting in a message like trigger not seen.  As a sanity check, I changed the trigger to be the rising edge of the alive_led, recompiled , committed to FLASH and powered the system on. This time in the SignalTap session, the "Read Data " operation returns the trace buffer with the trigger condition at the center indicating the trigger was hit.  If this is a case of the FPGA still configuring when RST# at the slot is deasserted, I'm not even sure delaying POST will help.   --- Quote End ---    It certainly sounds like your FPGA is not configuring within the required time.  Did you review the board design, i.e., how is the board configuring now, and can it ever meet timing? It might only be possible to meet power-on timing using CvP mode.    --- Quote Start ---   1. RST# may be required to initialize logic and it is never seen active when the FPGA logic is running.  As a confirmation, I would like to scope out the power ramp, RST# signal and the FPGA DONE signal or whatever indicated the FPGA is configed and running functional   clocks.  To test this , I figure I may be able to provide a local reset via a user switch so I can reset the FPGA manually. ( OR'ing the slot RST# with my on board reset )   --- Quote End ---   I have some notes on the timing requirements of PCIe;  http://www.ovro.caltech.edu/~dwh/wbsddc/ts4_power.pdf  PCIe link negotiation is supposed to start after RST# deasserts, so if your board is too slow, it fails to enumerate.     --- Quote Start ---   Altera has introducted CvP , in part I believe as the text below indicates ... past issues with power up timing vs FPGA configuration times. "....Altera’s new device configuration mode—configuration via protocol (CvP)—can be used with PCI Express® to configure the core fabric of Altera’s 28-nm Arria® V, Cyclone® V, and Stratix® V FPGAs. CvP can reduce product cost and board size, while simplifying the software usage model, and providing robust in-field system upgrade capability. In addition, the autonomous, embedded PCIe IP core helps ensure that designs meet PCIe power-up time requirements, irrespective of the FPGA core fabric configuration time, guaranteeing a wide range of interoperability with various PCIe-based computer platforms....."   --- Quote End ---   Yeah, that is the intention, but there are various errata for various chip revisions, so you'd have to see whether your board supports it.     --- Quote Start ---   2. Could the Intel PCIe chipset enumerate the PCIe slots only once and then disabled the slot ?  To investigate this I figure I will call HP support to see if they know anything about it.   --- Quote End ---   Its possible to re-enumerate the PCIe bus after power-on, however, this method is complicated by the fact that if the PCIe BIOS does not find a device at power-on, any of the bridges between the and the root-complex device might not be configured (their address map window might be too small). I'm not sure how various OSes handle re-enumeration, and whether that includes changing the PCIe bridge windows.  Keep in mind that Altera does not guarantee that their boards meet any sort of requirements with regards to meeting PCIe timing. Many people use these boards, and simply live with the fact that you have to warm-boot to get the board visible over PCIe ... this is one of those implied disclaimers that come with the name "Development Kit".   Personally I'd recommend getting another PCIe motherboard. I use a OneStopSystems motherboard and a x1 PCIe Expresscard to use my laptop to interface to PCIe boards. Its not the highest bandwidth solution, but its completely independent of any development machines.  http://www.onestopsystems.com/pcie_atx_bp.php  They also have an external PCIe enclosure (different web site name, but same company)  http://www.maxexpansion.com/  I haven't tried the cubes, they look cool though.  Cheers, Dave

[Altera_Forum]

Dave some random thoughts. 

 1. If I could isolate the +3.3 V and +12V PCIe slot power using a riser card, I could power the development card with its external ( brick ) power source.  This way it would be up and running when the slot gets reset ..  2. I need a POST / BIOS setting to delay the PCIe slot reset and enumeration but have not found such a delay ... the BIOS delay I set to 30 seconds today and it didn't help  3. For some reason, when the card is installed and I run PCIscope that is a utility to display the PCI configuration, a bus 02 shows up with nothing in it. If I force the bus   02, 00, 00 device to display resisters they are all F's and sometimes a Master Abort error is reported. I', thinking bus 02 must be a function of the present ( prst# )   being active . There is a X1, X4 and X8 I think and they say something about the card width capabilities. The wierd thing is they are souced by the FPGA and I would   think they would be TRI-state until the FPGA is configured. Tomorrow I will set the switch to isolate all 3 signals from the FPGA to see what happens.  4. I am still stumped on getting the system to "warm start" which I expect to be a start from the begining of POST but with the power not cycled.   This HP system seems to want to re cycle the power from off for restart and I don't want it to.  Thanks for your help ... it seems the development card , is a good candidate in this system to do some debugging when we thought it would be a cake walk ! I

[Altera_Forum]

--- Quote Start ---   4. I am still stumped on getting the system to "warm start" which I expect to be a start from the begining of POST but with the power not cycled.   This HP system seems to want to re cycle the power from off for restart and I don't want it to.   --- Quote End ---    Bob,   In most windows versions this is just a Start-> Restart  It's what I use when I've reprogrammed the board (but not flashed it) during development cycles.  I think the Jungo Windriver tools will see the board, even if the OS hasn't seen it during enumeration. They have an evaluation  version that might be worth trying.  Can you try any different (non HP) machine to see what happens? I use an earlier version of this....  http://www.overclockers.co.uk/showproduct.php?prodid=bu-188-ok&groupid=43&catid=2512&subcat=2685  ..with an old hard drive, PSU and monitor for testing. It's got XP with almost nothing installed so it boots pretty quickly.  I know Linux can be made to re-enumerate the PCI busses but I don't know anything about driving boards from Linux so that's not an option for me.   Nial

[Altera_Forum]

Might be worth trying to load a very minimal fpga image - maybe little more than a single PCIe BAR connected to an M9K block. 

 Another problem I've not seen a solution to is how to select a 'factory' image when using CvP, especialy if using AS to load the periphery image. There almost certainly isn't time to load a small image from the start of the EPCS which then determines whether to reload using either a 'factory' or 'user' image. If PS is used you can use a jumper to select the EPCS offset - not ideal, but better than having to use JTAG. Without that you always need the rest of the image in order to program a different one.

[Altera_Forum]

Thanks Nial, 

 On the HP Z220 ( Windows 7 ) I have been performing Start -> Restart ... seems to cycle the power off and restart .... this HP laptop ( Windows 7 ) says Start -> Restart will "close all programs , shutdown Windwos and restart windows .... That really isn't sufficient unless the Windows driver enumerates the PCI again. Older systems restarted POST keeping power on.  I don't really have other computers as the X86 system is a "reference system" . Soon I will have Linux up on another system and will have control at the device driver level since I work with the PCIe driver engineer.  Thanks for your help.  Bob.