VTune prints messages on all terminals of all users on one server?

[Guanfeng_Z_]

Yesterday, we installed the Sampling Driver for Vtune Amplifier.

Whenever I run the memory access, all users on team server receive following message:

Message from user@teamserver at Jun 14 14:10:15...

 kernel:Dazed and confused, but trying to continue

Message from user@teamserver at Jun 14 14:10:15 ...

 kernel:Uhhuh. NMI received for unknown reason 29 on CPU 11.

Message from user@teamserver at Jun 14 14:10:15 ...

 kernel:Do you have a strange power saving mode enabled?

Message from user@teamserver at Jun 14 14:10:15 ...

 kernel:Dazed and confused, but trying to continue

How to stop these prints? 

Thanks,

GZ

[Guanfeng_Z_]

System:

Linux version 3.10.0-123.el7.x86_64

Amplifier:

2018.1.0.535340 

[PAVEL_G_Intel · Accepted solution]

Hi,

The output you see isn't produced by VTune Amplifier. These are kernel messages. 
Your or system administrator may change kernel log level for suppression the messages.

- Pavel 

[Guanfeng_Z_]

Thanks for your reply, Pavel.

GZ

Pavel Gerasimov (Intel) wrote:

Hi,

The output you see isn't produced by VTune Amplifier. These are kernel messages. 
Your or system administrator may change kernel log level for suppression the messages.

- Pavel 

[McCalpinJohn]

We see these messages immediately after collection stops with the "memory-access" collection type (or from custom collection types derived from the "memory-access" collection).  In my testing they seem to occur a bit less frequently with 2018 Update 3 than with 2018 Update 2, but they don't disappear completely.   (My guess is that the driver unregisters the NMI interrupt handler before all of the logical processors have finished disabling the performance counter interrupt on overflow feature, but I can't follow the sep source well enough to understand control flow and ordering.)

If I understand the documentation correctly, the priority of these messages ("emerg") is the same as the priority of kernel "panic" messages, so you can't filter them based on priority.  The messages are sufficiently unique that they can be filtered based on their contents.

You can filter all of these by adding the following three lines to the beginning of the "rules" section in /etc/rsyslog.conf (then restarting rsyslogd):

:msg, contains, "Uhhuh. NMI received for unknown reason " ~
:msg, contains, "Do you have a strange power saving mode enabled?" ~
:msg, contains, "Dazed and confused, but trying to continue" ~

Because these messages can also occur due to real hardware errors, I recommend that you only include the second and third lines, and leave the single "NMI received for unknown reason" message. 

I am still trying to understand the rsyslog.conf syntax well enough to downgrade the remaining "NMI received for unknown reason" to a lower priority level, so it will be logged, but not broadcast.   Any rsyslog gurus out there?

[Guanfeng_Z_]

Thanks a lot, McCalpin.

GZ 

McCalpin, John wrote:

We see these messages immediately after collection stops with the "memory-access" collection type (or from custom collection types derived from the "memory-access" collection).  In my testing they seem to occur a bit less frequently with 2018 Update 3 than with 2018 Update 2, but they don't disappear completely.   (My guess is that the driver unregisters the NMI interrupt handler before all of the logical processors have finished disabling the performance counter interrupt on overflow feature, but I can't follow the sep source well enough to understand control flow and ordering.)

If I understand the documentation correctly, the priority of these messages ("emerg") is the same as the priority of kernel "panic" messages, so you can't filter them based on priority.  The messages are sufficiently unique that they can be filtered based on their contents.

You can filter all of these by adding the following three lines to the beginning of the "rules" section in /etc/rsyslog.conf (then restarting rsyslogd):

:msg, contains, "Uhhuh. NMI received for unknown reason " ~
:msg, contains, "Do you have a strange power saving mode enabled?" ~
:msg, contains, "Dazed and confused, but trying to continue" ~

Because these messages can also occur due to real hardware errors, I recommend that you only include the second and third lines, and leave the single "NMI received for unknown reason" message. 

I am still trying to understand the rsyslog.conf syntax well enough to downgrade the remaining "NMI received for unknown reason" to a lower priority level, so it will be logged, but not broadcast.   Any rsyslog gurus out there?