Cyclone V SoC Dev Kit PCIe End Port Example

[Altera_Forum]

I am trying to get the Cyclone V SoC Development Kit Board to behave as a PCIe End Port. I want to plug this board into a PCIe 4x slot on a PC motherboard running windows. This Dev Kit Board is really setup to be a root port and has a root port connector. So I can't plug the dev kit board directly into a slot on the motherboard. To solve this problem I purchased a male-to-male 4x adapter cable that also acts as a RX-to-TX crossover cable. A PCIe end port example design is included in the files I downloaded from Altera for the C5SoC dev kit. I had to make a few adjustments to the .qsf file, but I was able to get this end port example generated in qsys and compiled in quartus 13.1sp1. My problem is when I download the .sof into the dev kit fpga, the pc mother board doesn't seem to recognize it. I added a jtag master and a register to the design so I can read the PCIe LTSSM state bits. Using system console to read these bits, the PCIe controller appears to be going back and forth between the detect-quiet state and the polling-active state. My crossover adapter cable is fairly long (15"). I've ordered a shorter one (2") to see if this may be my problem. It will be a couple days before I get the 2" cable. In the mean time, I wanted to see if anyone else has tried to get the Cyclone V SoC Dev Kit Board to work as a PCIe End Port?

[Altera_Forum]

How are you getting the PCIe clock to your FPGA? 

 The PCIe connector has a 100MHz clock output ... but you need a 100MHz clock input to have the device work as an end-point.  Cheers, Dave

[Altera_Forum]

The Cyclone V SoC dev kit board has a PCIe clock generator device made by Silicon Labs. This clock generator has two differential 100 MHz clock outputs. One output is attached thru zero ohm resistors to the backplane connector and one is routed to the FPGA where it is used as the reference clock input to the PCIe high-speed transceivers. I removed the resistors that go from the clock generator to the backplane, since the pc motherboard also drives the PCIe REFCLK. So I don't think this is the problem and I don't think PCIe requires that the 100 MHz REFCLK come from the backplane. In fact Altera gives you the option to use a 125 MHz reference clock. 

 -kstolp

[Altera_Forum]

--- Quote Start ---  The Cyclone V SoC dev kit board has a PCIe clock generator device made by Silicon Labs. This clock generator has two differential 100 MHz clock outputs. One output is attached thru zero ohm resistors to the backplane connector and one is routed to the FPGA where it is used as the reference clock input to the PCIe high-speed transceivers. I removed the resistors that go from the clock generator to the backplane, since the pc motherboard also drives the PCIe REFCLK. So I don't think this is the problem and I don't think PCIe requires that the 100 MHz REFCLK come from the backplane. In fact Altera gives you the option to use a 125 MHz reference clock.   --- Quote End ---    PCIe in a motherboard expects a reference clock that is synchronous at all boards. That is why the SoC kit is supplying a reference clock to both the Cyclone V and the PCIe connector - because the device needs to see a synchronous reference clock.   Your setup needs to figure out a way to route the reference clock being driven onto the PCIe connector over to the FPGA. I'd recommend removing both 0-ohm resistors from the SiLabs part, and jumpering the PCIe clock coming into the connector, over to the resistor that drives the FPGA REFCLK pin.  Probe the pins with a scope to check that the differential clock at the FPGA looks clean.  Cheers, Dave

[Altera_Forum]

Dave, you are correct concerning REFCLK. It was the key to getting Altera's example end port design working on the Cyclone V SoC dev kit board. For fun, here is list of things I did to get this example to work: 

 1) Removed R249, R251, R253 & R254. Installed R250 & R252 (zero ohm jumpers). This makes the fpga pcie_refclk_p input come from the pcie backplane connector instead of the onboard oscillator.  2) Purchased a PCIe 4x male-to-male crossover cable to connect the Cyclone V SoC board to a pc motherboard. My cable came from Adex Electronics (www.adexelec.com), part number PE-FLEX4-CX-MM-15".  3) Used the ep_gen1x4_13_0_0_balanced design example found in the c:\altera\13.0\kits\cycloneVSX_5csxfc6df31es_soc\examples\pcie\endport_example directory after installing the kit files (cycloneVSX_5csxfc6df31es_soc_v13.0.0.1.exe) downloaded from Altera's website.  4) Made the following changes to the c5SoC_EndPoint.qsf file:  a) Added the following IO Bank VCCIO assignments. When I first compiled this example, VCCIO for several banks did not match the board. These lines fix that problem. Maybe these assignments are not necessary, but they match the board and make me feel better.   set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 3B set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 4A set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 6A set_global_assignment -name IOBANK_VCCIO 1.5V -section_id 6B set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 3A set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 5A set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 5B set_global_assignment -name IOBANK_VCCIO 2.5V -section_id 8A set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7A set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7B set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7C set_global_assignment -name IOBANK_VCCIO 3.3V -section_id 7D     b) I corrected the pin assignments for the user_led_fpga[3:1] pins, to be as follows:   set_location_assignment PIN_Y16 -to user_led_fpga[1] set_location_assignment PIN_W15 -to user_led_fpga[2] set_location_assignment PIN_AB17 -to user_led_fpga[3]     c) I changed the IO standard for the pcie_refclk input from "1.5V PCML" to "HCSL". I'm not sure this is necessary, but this is what the Cyclone V Hard IP PCIe User Guide says to do. My suspicion is that quartus ignores this assignment.  d) I changed the part number from "5CSXFC6D6F31C7" to "5CSXFC6D6F31C8ES". Quartus 13.0sp1 would not generate a programming file with the original "C7" part number. A message was generated indicated specifications for this device are subject to change. no programming file will be generated. I find it interesting that the design meets timing for coreclkout (125 MHz) with the "C7" part number, but not the "8ES" part number.   I hope this information helps someone else trying to use the Cyclone V SoC Dev Kit as a PCIe End Port.  -kstolp

[Altera_Forum]

--- Quote Start ---   Dave, you are correct concerning REFCLK. It was the key to getting Altera's example end port design working on the Cyclone V SoC dev kit board.   --- Quote End ---   Excellent!  Thanks for posting the modifications you needed to do.  Note that you can also create an AlteraWiki page, with a short write-up of what you did. That might be easier for the next user to locate, than this thread.  Cheers, Dave

[Altera_Forum]

So here is a couple more things I've learned: 

  

1. I was able to get the example end port design to meet timing with the "8ES" part number, by setting the routing timing optimization setting to "Maximum" in the more fitter settings window. If you make a new project from scratch, make sure to use the qsys and quartus settings found in Altera's example. These settings optimize for clock speed, which you need if you want to meet the 125 MHz requirement for coreclkout.  
2. Don't allow your BAR address space to become larger than 28-bits. This caused my PC not to boot, which took me a lot of time and frustration to figure out. Dave pointed this limitation out in an old thread titled "PCIe BAR size limitations in Qsys"  

  -kstolp

[Altera_Forum]

Did you check out the PDF in this thread? It has some comments about the settings required to meet timing (for Stratix IV and Cyclone IV devices); 

 http://www.alteraforum.com/forum/showthread.php?t=35678  Cheers, Dave

[Altera_Forum]

Dave, you seem to be some kind of PCIe expert. I appreciate your help. Thank you.

[Altera_Forum]

--- Quote Start ---  Dave, you seem to be some kind of PCIe expert.  --- Quote End ---   Nah, I'm just a guy who's had these types of problems himself :)    --- Quote Start ---  I appreciate your help. Thank you.  --- Quote End ---   You're welcome.  Cheers, Dave

[Altera_Forum]

Dave, 

 From the thread you posted above, you did some PCIe Hard IP compilation tests and had some issues with Altera's examples meeting timing. As mentioned in one of my previous posts to this thread, I've had some difficulty with the PCIe Hard IP meeting timing when targeting the FPGA on the Cyclone V SoC dev kit board. The main issue is the application layer clock has to be 125 MHz and the avalon switch fabric generated by Qsys can't really run at this clock speed unless the interconnects are kept to a minimum. The Altera example is a good starting point, but now I want to add some other components which will increase the number of avalon interconnects that need to run at 125 MHz. In fact, when I do this, it becomes impossible to meet the 125 MHz timing requirement.  My initial fix for this issue was to add clock bridges between all of the PCIe hard IP application layer interfaces and all the other avalon-mm components. This keeps the interconnect logic running at 125 MHz to a bare minimum. I chose to run the rest of the interconnect fabric at 50 MHz, which is an easy timing requirement to meet, no matter how complex my interconnect logic becomes. This solution seemed to be working fine, but today I encountered a problem. The PC I've been using works fine with my bridged design, but the PC we are using to develop the device driver on, does not recognize the board when my bridged design is loaded in the FPGA. Looking at the device manager (Windows 7), the PCIe root node connected the C5 SoC dev kit board, is unhappy. If I reload Altera's original example, both PC's recognize the board.  Is there a problem with my clock bridged approach? Any other thoughts as to what may be my problem?  Thanks, kstolp

[Altera_Forum]

Hi kstolp, 

 The example designs with the "Qsys PCIe core fails timing" thread used the PCIe Hard IP clock to clock the Qsys system primarily because that was how the first few examples I looked at implemented a simple design. In general I would not recommend doing this, but would instead use a local clock source, eg., a 100MHz Qsys clock should be fine. This allows you to implement a JTAG-to-Avalon-MM bridge into your Qsys system that will work regardless of the state of the PCIe clock.  The approach you are taking should work just fine, and would be the recommended approach.  There's a couple of things you can do;  1. Use a PCIe logic analyzer to trace the traffic that occurs when the system works and when it does not.  2. Use the SignalTap II logic analyzer to trace the traffic (if you do not have access to a PCIe analyzer)  3. Create a simulation and look to see whether you can "boot" your system, eg., enumerate the PCIe end-point and then access your Qsys system.  I'd recommend having (3) regardless of whether you do (1) or (2) as it forces you to really understand what needs to occur to get your system up-and-running. Once you have traces from (1) or (2), you can update your simulation to implement the same access sequence (eg., redundant accesses performed by the root-complex).  Cheers, Dave

[Altera_Forum]

Dave, Thanks for the advice. 

kstolp

[Altera_Forum]

Hi Dave, 

 My Question is a lil out of track but I would like you to ans this since You look Quite polished with PCIe related stuff.  I want to know If I can test PCIe Root Port without Connecting/attaching Endpoint with Linux Kernels.  Thanks Angad

[Altera_Forum]

Hi Angad, 

  --- Quote Start ---   I want to know If I can test PCIe Root Port without Connecting/attaching Endpoint with Linux Kernels.   --- Quote End ---   You mean you just want to see if the wires on the PCIe connector are working? If the root-complex is an FPGA, then sure, you can do whatever you like.  For example, you can test a high-speed transceiver by configuring it in BIST mode and have it generate a PRBS pattern, or you can use a transceiver IP core and configure it manually and have it output a desired pattern. For example, a square wave at a low enough frequency that you can see it on a scope. You can also loop back the TX and RX differential signals using a PCIe breakout cable (Samtec sell them).  http://www.samtec.com/technical-specifications/default.aspx?seriesmaster=pcrf  I use this type of cable (the one with the peripheral style connector) as a way to access transceivers on kits that do not have HSMC or FMC connectors.  Cheers, Dave

[Altera_Forum]

--- Quote Start ---  Hi Angad,  You mean you just want to see if the wires on the PCIe connector are working? If the root-complex is an FPGA, then sure, you can do whatever you like.  For example, you can test a high-speed transceiver by configuring it in BIST mode and have it generate a PRBS pattern, or you can use a transceiver IP core and configure it manually and have it output a desired pattern. For example, a square wave at a low enough frequency that you can see it on a scope. You can also loop back the TX and RX differential signals using a PCIe breakout cable (Samtec sell them).  http://www.samtec.com/technical-specifications/default.aspx?seriesmaster=pcrf  I use this type of cable (the one with the peripheral style connector) as a way to access transceivers on kits that do not have HSMC or FMC connectors.  Cheers, Dave  --- Quote End ---    Hi Dave,  Thanks for the quick response.  I am testing the following design: http://rocketboards.org/foswiki/view/projects/pcierootportwithmsi If you Look into "Open Issue" - FB 165881 – Kernel can’t boot up if don’t have end point  I have applied the patch that they are giving. It works fine on the Dev kit, But I tried the same patch for Other Cyclone V Device(5CSXFC6C6U23I7ES) The patch seems to fail.  Basically from the patch I understood that the RootPort is being forced to complete the Enumeration But I dont understand why it wont work for other device other than Dev kit.  Regards Angad

[Altera_Forum]

Hi Angad, 

   --- Quote Start ---   I am testing the following design: http://rocketboards.org/foswiki/view/projects/pcierootportwithmsi If you Look into "Open Issue" - FB 165881 – Kernel can’t boot up if don’t have end point  I have applied the patch that they are giving. It works fine on the Dev kit, But I tried the same patch for Other Cyclone V Device(5CSXFC6C6U23I7ES) The patch seems to fail.  Basically from the patch I understood that the RootPort is being forced to complete the Enumeration But I dont understand why it wont work for other device other than Dev kit.   --- Quote End ---    That is a terrible issue - its like saying "Sorry, your PC cannot boot if you do not have a card in the PCIe slot"!!!  Sorry, I've steered clear of the SoC devices for exactly this type of reason. I expected "teething" problems and didn't want to be the one to find them (been there, done that with earlier new generation devices).  I didn't read any of the code. Perhaps there is just a "bad decision" in the RTL or in the bootloader code that is the cause of this issue.  Cheers, Dave

[Altera_Forum]

--- Quote Start ---  Hi Angad,    That is a terrible issue - its like saying "Sorry, your PC cannot boot if you do not have a card in the PCIe slot"!!!  Sorry, I've steered clear of the SoC devices for exactly this type of reason. I expected "teething" problems and didn't want to be the one to find them (been there, done that with earlier new generation devices).  I didn't read any of the code. Perhaps there is just a "bad decision" in the RTL or in the bootloader code that is the cause of this issue.  Cheers, Dave  --- Quote End ---     Hi Dave,  Haha..that's exactly how I feel at this point of time when the Kernel hangs!!.. I have been seeking help from RocketBoards as well as Altera FAE but have not got any contextual solution to this issue. I don't know why would anyone make a device specific patch for a generic hard IP. Let me know if you get the time to go through the Patch and figure out a quick fix to the issue which I could try out.  Thanks Angad

[Altera_Forum]

Hi Angad, 

  --- Quote Start ---   I have been seeking help from RocketBoards as well as Altera FAE but have not got any contextual solution to this issue. I don't know why would anyone make a device specific patch for a generic hard IP. Let me know if you get the time to go through the Patch and figure out a quick fix to the issue which I could try out.   --- Quote End ---   Sorry, I won't get a chance to look at the patch. I recommending filing a Service Request directly with Altera (use the Altera web site) and see if they can help. At least you might get a chance to chat with the person who created the patch and see why it is needed and whether there is a better solution.  Cheers, Dave

[Altera_Forum]

Hello,  

  I am also facing the same issue "Kernel hangs after 'Starting kernel' ". Can you please message the link for the patch? I coudn't find rp_rtl.zip file in cv_soc_rp_full_design.tar.gz or cv_soc_rp_simple_design.tar.gz(As per solution for FB 165881 – Kernel can’t boot up if don’t have end point). Regards Kushal

[Altera_Forum]

--- Quote Start ---  Hello,    I am also facing the same issue "Kernel hangs after 'Starting kernel' ". Can you please message the link for the patch? I coudn't find rp_rtl.zip file in cv_soc_rp_full_design.tar.gz or cv_soc_rp_simple_design.tar.gz(As per solution for FB 165881 – Kernel can’t boot up if don’t have end point). Regards Kushal  --- Quote End ---    Hi Kaushal,  I am sure you would be using 13.1 or older version of Quartus. Use the latest version of Quartus 14.0 or higher. the issue has been addressed by Altera. For the higher version we do not require any patch hence Rocket boards have removed the patch from their site.  Regards Angad