Error message before machine crashes

[jburri]

We have been having problems with our machine crashing with no clear casue.  However, right before the machine crashes you see the following messages:

Jan  9 15:57:34 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:57:44 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:57:54 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:58:04 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:58:14 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:58:24 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:58:34 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:58:44 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:58:54 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:59:04 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:59:14 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:59:24 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:59:34 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:59:44 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 15:59:54 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 16:00:04 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 16:00:14 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 16:00:24 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 16:00:34 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 16:00:44 tacopie intelremotemond[12460]: Consumer: flush data to db
Jan  9 16:00:44 tacopie intelremotemond[12460]: Consumer: timeout, exiting...
Jan  9 16:00:44 tacopie intelremotemond[12601]: SendDataToBackend: path to agent is /opt/intel/ism/bin/intel64/intelremotemonagent
Jan  9 16:00:44 tacopie intelremotemond[12601]: SendDataToBackend: start agent
Jan  9 16:00:44 tacopie intelremotemond[12460]: Main: try to delete semaphore, status 0
Jan  9 16:00:44 tacopie intelremotemond[12460]: Main: daemon stopped, force is 0

I have not had much luck understanding what these message mean.  Does anyone have any insight?

I should add that you can compile and run the same code dozens of times without issue.  We are using icpc (ICC) 12.1.0 20111011 and CentOS release 6.3.

-Jeremy

[SergeyKostrov]

Hi Jeremy, It is not clear what your computer is doing just before it crashes. Do you compile some C/C++ project with Intel C/C++ compiler? Please specify. Also, Looking at these four log-messages: >>... >>Jan 9 16:00:44 tacopie intelremotemond[12601]: SendDataToBackend: path to agent is /opt/intel/ism/bin/intel64/intelremotemonagent >>Jan 9 16:00:44 tacopie intelremotemond[12601]: SendDataToBackend: start agent >>Jan 9 16:00:44 tacopie intelremotemond[12460]: Main: try to delete semaphore, status 0 >>Jan 9 16:00:44 tacopie intelremotemond[12460]: Main: daemon stopped, force is 0 >>... I see that some processing was done in Intel Remote Monitor Agent.

[jburri]

Hi Sergey,

The system is currently being used for the development of a couple of C++ program (one using only CPUs and another that is using CPUs and a GPU).  We are using icpc (ICC) 12.1.0 20111011.  Do you know anything about "Intel Remote Monitor Agent"? An initial goolge search didn't turn up anything useful for me.  We recently installed and started using VTune, but there does not seem to be a correlation between the running of VTune and the crashing of the system.  A lot of times the machine will become unresponsive in the middle of the night when no one is using.  

It has been a real head scratcher for us.

-Jeremy

[SergeyKostrov]

>>... We recently installed and started using VTune, but there does not seem to be a correlation between the >>running of VTune and the crashing of the system. Did VTune work when the system crashed? If Yes, I would try to uninstall VTune in order to verify that it is not involved in these craches.

[SergeyKostrov]

And one more thing: >>...SendDataToBackend: path to agent is /opt/intel/ism/bin/intel64/intelremotemonagent I wouldn't say that there is no relation with some Intel software.

[Bernard]

@Jeremy

Can you enumerate all active daemons on your compute before the crashr?

>>>Jan 9 16:00:44 tacopie intelremotemond[12460]: Main: try to delete semaphore, status 0>>>

Could that be somehow relate to synchronization issue?

[Brandon_H_Intel]

Hi jburri,

This intelremotemonagent service is part of the Intel(R) Software Improvement program functionality.  I've had one of the developers of this take a look and he doesn't see anything unusual about the logging messages your provide.  We're working on what more information could help us determine if there is anything unusual here - there is a log (intelremotemonagent.<uid>.log) but it's in /tmp which would get wiped after reboot.  I'll let you know as soon as we have anything further.

[Brandon_H_Intel]

It sounds like you can configure /tmp in your OS to not have all files in it deleted after reboot.  If you could do that, and if you get another crash, send us the log file, that would be helpful.

[SergeyKostrov]

>>...This intelremotemonagent service is part of the Intel(R) Software Improvement program functionality... Brandon, Here is a question: How a user can disable or turn it off to verify that all the rest software is not creating that problem?`

[a_kaliazin]

Hi all It looks like the Sergey's last plea was left unanswered, so I'd like to raise this question again - how can I turn off completely this annoying service which we never have asked for? For me it only fills up the syslog without improving anything. VTune keeps crashing, then suggests to send a report and crashes at that as well.

[Mark_S_Intel1]

You can disable the service by running ISM in /opt/intel/ism or ${HOME}/intel/ism folder, go to settings, click "Custoemr Feedback Options" button and choose "Manual launch only".  The executable name is "ism".

 --mark

[Dogan_Y_]

There is an uninstall script under /opt/intel/ism directory in linux. In order to disable the Intel Software Manager files on your system, you can run it.