Cyclone IV Transciever Starter Kit Automation Issue MAX_error LED stuck on.

[Altera_Forum]

We have an automated test system running with Cyclone IV Transciever Cards acting as PCIe EP's. 

 Our automation system load the .elf to the card memory via a NIOS console running on a host PC.   The Host PC is on all the time , but the DUT is power cycled where the PCIe slot power is removed and  re-applied at times ... Is this legal ... since the Host PC is supplying +5V USB to the USB I/F device but the other supplies to that device are off .... ?  At some time, unknown, the Cyclone card red LED , MAX_error comes on and will not clear when the power is  removed from the Cyclone card , unless the USB cable , supplying +5V USB is removed also.  I'm thinking the USB I/F device may be in some error state and will not clear unless the +5V USB power is removed .  With this information, does anyone have ideas on:  1. How what this condition is ? 2. How to avoid this condition OR clear this condition from the Host PC without shutting down the PC or removing the USB cable ?  Thanks, Bob.

[Altera_Forum]

--- Quote Start ---   The Host PC is on all the time , but the DUT is power cycled where the PCIe slot power is removed and  re-applied at times ... Is this legal ... since the Host PC is supplying +5V USB to the USB I/F device but the other supplies to that device are off .... ?   --- Quote End ---   This is unlikely to be the source of your issue. If you look at the schematic, the FTDI device is the only component powered from 5V. If the board was in stand-alone mode, then it still needs an external 12V power supply.  I just plugged my kit in with USB and power, and then repeatedly plugged and unplugged the 12V connector without turning the power switch off, and I could not get it to fail.    --- Quote Start ---   At some time, unknown, the Cyclone card red LED , MAX_error comes on and will not clear when the power is  removed from the Cyclone card , unless the USB cable , supplying +5V USB is removed also.  I'm thinking the USB I/F device may be in some error state and will not clear unless the +5V USB power is removed .  With this information, does anyone have ideas on:  1. How what this condition is ? 2. How to avoid this condition OR clear this condition from the Host PC without shutting down the PC or removing the USB cable ?   --- Quote End ---    Well, the first thing is to look at the schematic and find the red LED ... p10 has a CONF_DONE_LED ... is that the one?  The fact that CONF_DONE did not assert, means that there was a configuration issue at power-on. Perhaps the external power source is not ramped up cleanly.  Here's a potential solution;  1. Trigger the condition where the red LED comes on.  2. Press the PGM_CONF button on the board.  Does the FPGA configure correctly?  If yes, then this tells you that the board is powered correctly, but the power-on sequence has some form of issue that causes configuration to fail. Your solution will be to issue a configuration request via USB.  How do you do that? Well, the kit has a MAX II system controller, and that typically has a Virtual JTAG component instantiated to provide access to some internal registers ...  Here's what I see from the kit I have ...  $ jtagconfig -n 1) USB-Blaster 028010DD EP4CGX15 Node 19104600 Nios II# 0 Node 0C006E00 JTAG UART# 0 020A40DD 5M(1270ZF324|2210Z)/EPM2210 Node 00406E00 (110:8)# 0   The second device contains a Virtual JTAG core.  Look at page 14 of the reference manual and you'll see a block diagram of the system controller.  Unfortunately the system controller is not well documented in the reference manual, but the source code is available.  So "all you need to do" is read the MAX II source code, figure out the register map of the device, then you can issue a Virtual JTAG command to pulse nCONFIG.  *Alternative solution*  1. Trigger the condition where the red LED comes on.  2. Scan the JTAG chain.  If you can see the Cyclone IV GX device, then you can issue a JTAG command to it to pulse nCONFIG. I have some code I can dig up that shows how to do that.  Cheers, Dave

[Altera_Forum]

Thanks Dave ...Yes to page 10 of the schematic , but the red LED is an error indicator and I believe the RED_LED is D1 and associated with MAX_ERROR. Given that, it seems to indicate a load error , and I can trigger on that but this error is pervasive across power cycles until the USB cable is removed ... to me this indicates that the USB cable is holding U5 in some state that causes the U10 MAX II ( EPM2210 ) to flag an error ... possibly not an actual load error but some error ... i would need to check the MAX II code to see what all the sources are of the MAX_ERROR...  

 I suspect this is a power sequencing issue ... since the emulator I am connected to does alot of power on/ off cycles and I believe there may may be an issue there or else with the automation that runs overnight since when I am manually working at the emulator and power cycle off and on ... I don't see any issues.  I have asked to capture a screen shot of the NIOSII command terminal that is attempting to load a .elf and in this error state, gets an error indicating no USB Blaster is present.  Thanks, Bob.

[Altera_Forum]

--- Quote Start ---  Yes to page 10 of the schematic , but the red LED is an error indicator and I believe the RED_LED is D1 and associated with MAX_ERROR.   --- Quote End ---   Oh yeah, I see, I associated the wrong signal with the RED LED.  Either way, my solutions are what you should test out.  If the error can be cleared by pressing the PGM_CONF button, then you should be able to do something equivalent via USB.    --- Quote Start ---   Given that, it seems to indicate a load error,   --- Quote End ---   And if you read the source for the MAX II design, you'd see why it was asserting the LED signal.    --- Quote Start ---   and I can trigger on that but this error is pervasive across power cycles until the USB cable is removed ... to me this indicates that the USB cable is holding U5 in some state that causes the U10 MAX II ( EPM2210 ) to flag an error ... possibly not an actual load error but some error ... i would need to check the MAX II code to see what all the sources are of the MAX_ERROR...    --- Quote End ---   Exactly :)    --- Quote Start ---   I have asked to capture a screen shot of the NIOSII command terminal that is attempting to load a .elf and in this error state, gets an error indicating no USB Blaster is present.   --- Quote End ---   Ask them to issue the jtagconfig -n command I used above. If the MAX II still shows up, then that means the JTAG chain is still good, and you could either access the MAX II registers to issue a command, or access the Cyclone via JTAG and tell it to reconfigure.  Cheers, Dave

[Altera_Forum]

Dave and others, 

 I have attached some NIOS Command Window results in the hang state and not hang state.  Here is the output.  Issuing nios2 flash command: "C:\altera\13.1\nios2eds\Nios II Command Shell.bat" nios2-download -r -g > %TEMP%\nios2-download.txt there are no nios ii cpus with debug modules available which match the values specified. please check that your pld is correctly configured, downloading a new sof file if necessary.  This kind of tells me that the FPGA did not load successfully after a power cycle of the system.  I tried pressing the PGM_CONF in the error state and the error LED goes off then on again.  I have a log of automation results and apparently a card that is hung with the Error LED on had just previously passed, indicating the main thing was probably a power cycle ... Now I am wondering if somehow the USB "connection was possibly active from the PC to the USB controller on the Cyclone card , when the Cyclone card was re-cycled via the PCIe  slot power being cycled off then on ... In this case, the USB controller didn't get a power cycle since it is held up via the USB +5VDC from the host PC . I will try that experiment to see if I get the same behavior .  Thanks, Bob.

[Altera_Forum]

This picture depicts the depicts the Cyclone card in the error state.  

Note the Blue power LED on, a Green config LED on and the RED max_error LED on.

[Altera_Forum]

Hi Bob, 

  --- Quote Start ---   I have attached some NIOS Command Window results in the hang state and not hang state.   --- Quote End ---   The NIOS tools require communication with a *configured* FPGA. Your comments above already established that configuration failed, so its no surprise that a NIOS tool will not work.   This is why I requested you run jtagconfig -n, as this would establish whether the JTAG chain was still viable in the error state. If it is, then you might be able to automate the recovery.    --- Quote Start ---   I tried pressing the PGM_CONF in the error state and the error LED goes off then on again.   --- Quote End ---   Hmm, that is weird. If the red LED comes back on *again* that would imply the MAX II does not like some hardware pre-condition for loading the FPGA. Did you look at the MAX II code?  Can you please run jtagconfig -n when the red LED is on, and see whether the JTAG chain is still detected ok, thanks!  Cheers, Dave

[Altera_Forum]

Solved .....  

 I took a Cyclone card to my office, connected it to USB on my Laptop and ran nios2-terminal from the NIOS Command Shell and got a connection.  Next, I powered down the Cyclone card and then powered back on using the brick power and the card power slide switch. The resulting behavior is a red MAX_ERROR LED. I will tell the automation team not to do it or to somehow release the connection before cycling the Cyclone PCIe slot power. .  I tried this by terminating the nios2-terminal session with CNTL-C and in that state, the card can be power cycled without disconnecting the USB cable +5 VDC and the FPGA is configured without any issue.

[Altera_Forum]

--- Quote Start ---   Solved .....    --- Quote End ---   Not so fast ... :)    --- Quote Start ---   I took a Cyclone card to my office, connected it to USB on my Laptop and ran nios2-terminal from the NIOS Command Shell and got a connection.  Next, I powered down the Cyclone card and then powered back on using the brick power and the card power slide switch. The resulting behavior is a red MAX_ERROR LED. I will tell the automation team not to do it or to somehow release the connection before cycling the Cyclone PCIe slot power. .   --- Quote End ---   Just to avoid confusion on my part, can you run through this sequence without using nios2-terminal?  I just plugged in my card, and no combination of power cycling, unplugging the USB connector, or power cable with or without toggling the power switch off causes a red LED to come on.  If you can establish that its the nios2-terminal connection that causes the issue, then you can probably disconnect, power-cycle, and then reconnect.  Cheers, Dave

[Altera_Forum]

Hi Dave, 

 This seems un-scientific ... I tried to re-produce the result seen before lunch with the automation folks and was not able to reproduce the result. ie no red MAX_EROR LED.  Does that mean the Cyclone card was in some state that when the nios2-terminal was connected and the card power cycled caused the MAX_ERROR LED to come on ...  Just 1/2 an hour ago I saw another Cyclone card locked up with the MAX_ERROR LED stuck on and in that case the NIOS Eclipse GUI was active not a NIOS Command terminal ... so I agree I need to work some more to get to root cause ... all I can say was that ... whatever I did in my office environment ... I got to a matching behavior as seen in an automation environment.  Let me gather some more info on this and modify the status to .... "trending to solved ....."   Thanks, Bob.

[Altera_Forum]

--- Quote Start ---   This seems un-scientific ... I tried to re-produce the result seen before lunch with the automation folks and was not able to reproduce the result. ie no red MAX_EROR LED.   --- Quote End ---   The un-scientific term is that you have a Heisen-bug, i.e., a bug that disappears when observed :)    --- Quote Start ---   Does that mean the Cyclone card was in some state that when the nios2-terminal was connected and the card power cycled caused the MAX_ERROR LED to come on ...   --- Quote End ---   Its hard to say until you can duplicate the error systemically on your board, and then on another. Only then do you know you're not dealing with a mildly damaged board.    --- Quote Start ---   Just 1/2 an hour ago I saw another Cyclone card locked up with the MAX_ERROR LED stuck on and in that case the NIOS Eclipse GUI was active not a NIOS Command terminal ... so I agree I need to work some more to get to root cause ... all I can say was that ... whatever I did in my office environment ... I got to a matching behavior as seen in an automation environment.  Let me gather some more info on this and modify the status to .... "trending to solved ....."    --- Quote End ---   If you could determine whether the JTAG chain is still valid when the error occurs, that would be a useful piece of information :)  Good luck!  Cheers, Dave

[Altera_Forum]

Still monitoring this ... I am going to scope the PCIe slot +12V_PCIE and the +3.3V_PCIE and capture them in infinite persistence mode to check for anything weird.