Questa simulation hangs (nondeterministic)

[c-thaler]

I run a Questa simulation for an IP core from the command line on a Linux server. Usually, the simulation finishs successfully. But every now and then the simulation hangs.
When it hangs, I can see that the CPU load is 100% but I can't break the simulation by hitting Ctrl-C. The key-press is detected ("# Break key hit") but the simulation won't halt.

Since I use the same seed for random number generation in each run, this seems like nondeterministic behavior.

I have no clue, where this behavior is comming from. Are there any command line options that I can use to get closer to the problem? How can I go one with debugging in such a case?

[c-thaler]

I tried to run all processes with valgrind. This causes the simulation to terminate with:

"Process terminating with default action of signal 41 (SIGRT9)"

and

"Trouble with Simulation Kernel."

Of course, this might be a side effect of valgrind.

[c-thaler]

I tried a testbench from another project with valgrind and it works fine. So it seems there is really something wrong with the simulator.

[c-thaler]

I was able to reproduce the hang up on a Windows system.
In contrast to the Linux system I was able to stop the process with Ctrl-C leading to the following output:

vish lost connection to vsim process.

                                                                                                                                                                                    
simulator: Unexpected EOF on RPC channel sock0000000007A71780
Kernel lost connection to front end process.
** Fatal: Exiting VSIM license process.

[TingJiangT_Intel]

As it worked fine after you changed the project, the error may related to the project and testbench. It may be that your project got stuck during simulation, causing excessive CPU resource usage.

You may check your testbench to see if there are any operations that cause an infinite loop.

[c-thaler]

Thanks for the reply.

Questa is able to detect zero delay loops, so I should see an error about this. Furthermore, most of the time the tests run just fine. I have to run a single test a few hundred times to cause this behavior. That means it is nondeterministic behavior. If there is an infinite zero delay loop I should see the hang up in every simulation run.
Besides that, I should be able to break/pause simulation in such a loop, which does not work either.

[c-thaler]

I was able to reproduce with the other project, too. It took a few thousand simulation runs over the weekend. But finally it got stuck with 100% CPU load.

[TingJiangT_Intel]

I see, can you upload your project and tell me your detail operation steps to help me reproduce the issue?

[c-thaler]

The whole project is quite huge. I will try to reproduce the issue with a smaller example, e.g. a module test. This may take a few days.

[TingJiangT_Intel]

Ok, really appreciate for this.

[c-thaler]

I'm finished with the smaller example. How can I upload the shipment without making it available to the public?

[TingJiangT_Intel]

Hi there, I have sent you an email, you can upload your project via it.

[TingJiangT_Intel]

Hi there, I noticed that your email has been blocked. Seems it contains some files that is not allowed.

Please check the attachments. You can upload qar file after archived.

[c-thaler]

Thanks for he response. I'll try it again.

Unfortunately, I cannot build an qar file since it is not an Quartus project. All Questa call are made from a Shell Script. The mail was probably blocked because of this script, since it is executable.

[TingJiangT_Intel]

Hi there, is there any updates on this issue?

[c-thaler]

Hi,

I've sent you another mail on 07-19 with a reproducible example attached. Was it blocked again?

[c-thaler]

I've sent you another mail with a link to our file exchange server, where I've just uploaded the reproducible example.

[TingJiangT_Intel]

Hi there, I checked your script. It seems that you are not using the officially generated scripts. Using non-official scripts can lead to unpredictable behavior during simulation. It is recommended to use the simulation scripts generated by Quartus. For more details, please refer to the following link:

https://www.intel.com/content/www/us/en/docs/programmable/730191/24-1/generating-a-msim-setup-tcl-simulation.html

[c-thaler]

In my case, there are no generated scripts since I'm not using any Intel IP in the simulation. It is just a testbench for a simple module.

[TingJiangT_Intel]

Yes, but seems you write many scripts yourself, we can't make sure if there is any issue related to these flow. With the generated script it will ensure all the necessary file is included and the flow was verified.

[c-thaler]

Well, the script and the flow work 99.9% of the time. But every now and then I see a hang up of the simulator, without doing any changes to the VHDL files or the flow at all.
How can this behavior be caused by missing files or a bad flow? I mean, simulation is deterministic. I'd expect the same outcome on every run.