Different results on hardware compared to simulator

[Altera_Forum]

I'm having trouble when I run my code on the actual Cyclone III FPGA. In the simulator it works fine. I have set up state machines to make sure that clock signals to registers are occurring after plenty of data setup time has passed. I'm running the chip with just a 14 MHz clock so its not very fast. 

 In ModelSim everything works perfectly. My code is modular so I am building up the final code one module at a time. I can put in several modules, test, and everything works as expected. But at some point when I add in another module everything starts to go wrong. Modules that worked earlier no longer work.  I added in some ports on the modules to bring out the data to some external LEDS so I can see what the signals are doing. Often that causes the code to just start working again properly. The results are very inconsistent as I make changes to the code.  What am I missing here that makes the code work incorrectly on the actual hardware?

[Altera_Forum]

--- Quote Start ---  We are using a TS-7400 CPU board which basically has a 20 bit data IO bus that comes off of a CPLD on their board. They use a EP9302 processor chip which is running from the 14.7456 MHz crystal. However, I do not know about whats happening internal to that processor. The 9302 talks to their CPLD and manipulates the data IO bus. They also pass the 14 MHz clock signal through their CPLD (which ends up being the FPGA clk) and they can turn it off or on in the CPLD (which means there must be delays as it passes through their CPLD).   --- Quote End ---  I looked at the TS-7300 a while back. There was an OpenCores.org design for the external bus interface. If Technologic Systems have used the same design for the TS-7400, you might get some nice tips in there. The code was written bu Jesse Off, one of TS's design engineers.  So your FPGA is attached to their board? Is this a PC104 module, or something like that?    --- Quote Start ---   We have decided to use 8 bits as a multiplexed addr/data bus and 3 other lines as ale, rd, and wr. Because of the number of IO lines we need from the FPGA its not practical to use more lines to interface to the TS7400.  So our code that runs on the TS7400 basically goes through this process: Set the Data Direction Register (DDR) for DIO[7:0] to output. Place the data on the data bus. Set the ale (or wr) bit high. Set the ale (or wr) bit low.  Read cycles would by like this: Set the DDR for DIO[7:0] to input. Set the rd bit high. Read the DIO[7:0] data Set the rd bit low.   --- Quote End ---  So you have essentially taken the external processor bus and turned it into a bit-banged I/O bus. This is not exactly the most efficient way you could have done this, but lets put that aside for now, and get you something that will work. We can 'optimize' later.    --- Quote Start ---    By scope measurements I have determined that the rate that we can go through this process takes about 1us for each steo (depending on what instructions we need to execute). That means when writing (addr or data) the data will be on the DIO bus about 1 us before the rising edge of ale or wr and will be held there until about 1 us after the falling edge of ale or wr. We would read data in about 1 us after asserting rd. Seems like plenty of setup and hold time to me though the ale, wr, and rd are not necessarily synchronous with the clk signal.   --- Quote End ---  Right, this is the second option I mentioned, and Nial expanded upon.  You are interfacing to an asynchronous bus, and the timing of that bus is slow compared to your FPGA clock (which with the use of a PLL can be ~100MHz).   To interface to this type of bus, you need to:  1) Use synchronizers (a chain of at least two DFFs) to synchronize the control signals from the TS-7400 board (ALE, RD and WR, but not the 8-bit data bus) to the FPGA clock domain.  2) You create an edge detector to detect when ALE pulses, and use that to capture the address into a register clocked by your FPGA clock.  3) You create a state machine that is triggered by the ALE pulse (in the FPGA clock domain), and use that to sequence the read or write cycle from the FPGA.  If I was writing this code, I'd create a bus functional model (BFM), which is just a fancy way of saying a simulation for the read/write accesses. I'd then simulate the logic in Modelsim to make sure I could read and write from some registers in my FPGA. Then I'd run some hardware tests and capture some SignalTap II traces to confirm that my BFM signals matched the hardware. Because your signals are all asynchronous, your .SDC file will cut the timing along those signal paths.  If this all sounds completely foreign to you, let us know, and Nial or myself, or someone else will give you code examples.  Cheers, Dave

[Altera_Forum]

--- Quote Start ---  I looked at the TS-7300 a while back. There was an OpenCores.org design for the external bus interface. If Technologic Systems have used the same design for the TS-7400, you might get some nice tips in there. The code was written bu Jesse Off, one of TS's design engineers.  So your FPGA is attached to their board? Is this a PC104 module, or something like that?  So you have essentially taken the external processor bus and turned it into a bit-banged I/O bus. This is not exactly the most efficient way you could have done this, but lets put that aside for now, and get you something that will work. We can 'optimize' later.    --- Quote End ---   The TS7400 does not use a PC104 interface. It has this 20 bit DIO bus coming off their CPLD going to a 40 pin header (there are other useful signals in there like a SPI interface and a uart interface, but that is not interesting to our discussion). You can't connect to the cpu directly (which would have been way more efficient). As an aside, my next project is to design our own cpu board and then we can get much more efficiency.    --- Quote Start ---   You are interfacing to an asynchronous bus, and the timing of that bus is slow compared to your FPGA clock (which with the use of a PLL can be ~100MHz).   To interface to this type of bus, you need to:  1) Use synchronizers (a chain of at least two DFFs) to synchronize the control signals from the TS-7400 board (ALE, RD and WR, but not the 8-bit data bus) to the FPGA clock domain.  2) You create an edge detector to detect when ALE pulses, and use that to capture the address into a register clocked by your FPGA clock.  3) You create a state machine that is triggered by the ALE pulse (in the FPGA clock domain), and use that to sequence the read or write cycle from the FPGA.  If I was writing this code, I'd create a bus functional model (BFM), which is just a fancy way of saying a simulation for the read/write accesses. I'd then simulate the logic in Modelsim to make sure I could read and write from some registers in my FPGA. Then I'd run some hardware tests and capture some SignalTap II traces to confirm that my BFM signals matched the hardware. Because your signals are all asynchronous, your .SDC file will cut the timing along those signal paths.  If this all sounds completely foreign to you, let us know, and Nial or myself, or someone else will give you code examples.  Cheers, Dave  --- Quote End ---    My original efforts in designing for programmable logic started only about 2 months ago. I learn Quartus, Verilog, SystemVerilog, and how to write testbenches. At the time I thought timing would not be a factor because of the slow speeds. I can see now that timing is everything, it does not solve itself automatically, even in a simple slow design.  I had written testbenches for each of my modules all the way up to the final top level testbench. It does exactly what you suggest, emulates the bus, writing addresses, writing data, reading data and using approximately the same timing as I discovered on the scope. So the functional models have been working from the very early days, according to ModelSim. Not having a problem with that. When running on the FPGA then I discovered the problems with timing and registers not doing what ModelSim said they should.  I did in fact write a state machine for the main IO at the top level and it does what you suggested. It basically captures the address in a register during the ale pulse (I have been trying different suggestions on that to use it to gate the main clock so that all timing is based on the one 14 MHz clock). Those function correctly in the test bench.  I'm coming up to speed on TimeQuest and SDC files and the Timing Advisor. Following its suggestions I have been able to get the slack time to be positive in all but one signal and that one is off by only a couple picoseconds. but I have not tried that in the hardware yet.

[Altera_Forum]

--- Quote Start ---  The TS7400 does not use a PC104 interface. It has this 20 bit DIO bus coming off their CPLD going to a 40 pin header (there are other useful signals in there like a SPI interface and a uart interface, but that is not interesting to our discussion). You can't connect to the cpu directly (which would have been way more efficient).   --- Quote End ---  Ok. Your bit-banged bus proposal makes sense given the hardware limitations.  The lesson to take from this experience is to define the interfacing requirements before buying any hardware :)    --- Quote Start ---   As an aside, my next project is to design our own cpu board and then we can get much more efficiency.   --- Quote End ---  Since that particular design is not on-topic for the Altera Forum, email me off-list with your requirements and thoughts, and I'll give you feedback.    --- Quote Start ---   My original efforts in designing for programmable logic started only about 2 months ago. I learn Quartus, Verilog, SystemVerilog, and how to write testbenches. At the time I thought timing would not be a factor because of the slow speeds. I can see now that timing is everything, it does not solve itself automatically, even in a simple slow design.  I had written testbenches for each of my modules all the way up to the final top level testbench. It does exactly what you suggest, emulates the bus, writing addresses, writing data, reading data and using approximately the same timing as I discovered on the scope. So the functional models have been working from the very early days, according to ModelSim. Not having a problem with that. When running on the FPGA then I discovered the problems with timing and registers not doing what ModelSim said they should.  I did in fact write a state machine for the main IO at the top level and it does what you suggested. It basically captures the address in a register during the ale pulse (I have been trying different suggestions on that to use it to gate the main clock so that all timing is based on the one 14 MHz clock). Those function correctly in the test bench.  I'm coming up to speed on TimeQuest and SDC files and the Timing Advisor. Following its suggestions I have been able to get the slack time to be positive in all but one signal and that one is off by only a couple picoseconds. but I have not tried that in the hardware yet.  --- Quote End ---  Excellent effort!  If you can post a zip file with your code and testbenches, then please do. If you're not sure you want to post that material to the Altera forum, then email them to me, and direct me to the parts of the code you are unsure about, and I'll take a look (my forum name is my email address).  Cheers, Dave

[Altera_Forum]

--- Quote Start ---  Ok. Your bit-banged bus proposal makes sense given the hardware limitations.  The lesson to take from this experience is to define the interfacing requirements before buying any hardware :)  Cheers, Dave  --- Quote End ---    Yeah, but the the choice of the ts7400 hardware was pretty much a done deal when we started this effort. We had several other interface boards designed already that use the TS7400. I got into CPLD and then FPGA because we needed to design our own UARTS and a DMX512 lighting controller. The Cyclone III FPGA with its RAM inside makes a great solution for a DMX serial port (which has to send out the same set of 512 bytes many times per second. No UART chip available has that type of operation where you can just load up the data you want and it continually sends it out.  Anyway, you have been a great help, along with a number of others. I will keep posting as necessary and perhaps take you up on your offer to look at my code.

[Altera_Forum]

--- Quote Start ---   Anyway, you have been a great help, along with a number of others. I will keep posting as necessary and perhaps take you up on your offer to look at my code.  --- Quote End ---    Good luck with your project.  Cheers, Dave