Analyzers
Talk to fellow users of Intel Analyzer tools (Intel VTune™ Profiler, Intel Advisor)

Profiling an application which uses SIGNALS

Wolfgang_K_
Beginner
1,793 Views

Hello,
we are using Intel VTune 2015 for profiling our application which is running under CentOS 5.11.
Our application uses c++ signals for the control flow. When trying to do a basic hotspots analysis using amplxe-cl command line tool with the following parameters: 
-duration 20 --run-pass-thru=--profiling-signal=1
VTune yields the following error message when detaching after the 20 seconds duration. Alternative to the number 1 I also tried number 4 without any change in results.

amplxe: Error: Assertion failed: handler_ex1445: obj->is_first_class_handler_set[signo] == 1 : BUG! : signo == 1. Please contact the technical support. 

Without using "--run-pass-thru=--profiling-signal=1" as parameter the signal handlers of our application do not work after profiling and the process is ended when receiving a signal.
Please provide guidance on how to use VTune in this scenario.

Best Regards,
Stephan

0 Kudos
28 Replies
Peter_W_Intel
Employee
1,378 Views

As default, VTune Amplifier will use application's alter signal stack which is 64KB limited. So, you may not use VTune's alter signal stack from application's. For example:

amplxe-cl –collect hotspots –run-pass-thru=-no-altstack -- application

0 Kudos
Wolfgang_K_
Beginner
1,378 Views

Hello Peter,
thanks for your response. We have been trying the solution you have proposed.
Unfortunately the result for us does not change. When our application receives a signal after VTune is detached the process is just shut down without our handlers taking care of the signal.
Please keep in mind that we do not set up an alternate signal stack using function sigaltstack.

We do the following to set up the signal handlers on our side.
- Add signal handlers using sigaction
  We use sigfillset on our sigactions to ignore all other signals while a handler is running
  The sa_flags are set to SA_RESTART
- For SIGINT, SIGQUIT, SIGTERM, SIGSEGV, SIGFPE, SIGILL, SIGABRT we set handler functions
- SIGHUP, SIGCHLD, SIGPIPE, SIGALRM, SIGXFSZ, SIGSTKFLT, SIGTSTP, SIGTTIN, SIGTTOU, SIGURG, SIGXCPU, SIGVTALRM, SIGWINCH, SIGPWR, SIGIO we set to SIG_ING
- We use pthread_sigmask(SIG_SETMASK, &SignalSet, NULL) with 
  sigset_t SignalSet;
  sigfillset(&SignalSet);
  sigdelset(&SignalSet, SIGINT);
  sigdelset(&SignalSet, SIGQUIT);
  sigdelset(&SignalSet, SIGTERM);
  sigdelset(&SignalSet, SIGUSR1);
  sigdelset(&SignalSet, SIGUSR2);
  to block all signals except for the ones needed.
- When receiving a signal other that SIGUSR2 in our handler, we set the handler for the above signals to SIG_IGN to ignore further signals

Therefor we were trying to use "--run-pass-thru=--profiling-signal" as we though this would tell VTune to use a signal which is not used on our side, hoping to preserve our signal handlers.

Could you please provide further details on the usage of signals with VTune and how to handle the issue on our side?

Best Regards,
Stephan

0 Kudos
Peter_W_Intel
Employee
1,378 Views

Is it possible that you can provide vtune's result directory for investigating?

I have escalated this issue to the developer, please feel free to send me private message with attached vtune result. Thank you.

0 Kudos
Peter_W_Intel
Employee
1,378 Views

I got comment from developer - "Could you please ask the customer to try to pass any unused signal from SIGRTMIN..SIGRTMAX range to --profiled signal option?"

0 Kudos
Wolfgang_K_
Beginner
1,378 Views

Thanks for the input. We tried the --profiling-signal option using a signal between SIGRTMIN..SIGRTMAX with no improvement.
We managed to implement a small testapplication having the same signalhandling as our real application. Please find it attached to this post. Maybe a developer on your side can use it to reproduce the error and provide input on how to fix the issue.
Please perform the following steps for reproduction:

extract: tar -zxf SignalsTest.tar.gz
go to SignalsTest/Debug
run: make all
run: 
  ./SignalsTest
  kill -12 <PID of SignalsTest>
  kill -2 <PID of SignalsTest>
  kill -9 <PID of SignalsTest>
result:
  Starting
  Caught 12.
  Caught 2.
  Killed

run: 
  ./SignalsTest
  kill -12 <PID of SignalsTest>
  ./amplxe-cl -collect hotspots -duration 10 -target-process SignalsTest
  kill -2 <PID of SignalsTest>

result:
  Starting
  Caught 12.

Process is killed when receiving signal 2 as handler is broken after VTune analysis.

0 Kudos
Peter_W_Intel
Employee
1,378 Views

Thanks for you file, I have forwarded this to the developer.

It is vacation season for now - the response might be a little bit slow. Thanks for your understandings:-)

0 Kudos
Peter_W_Intel
Employee
1,378 Views

Just let you know, the developer confirmed that this problem can be reproducible...it was under investigating. The solution will be slow due to Holiday. Merry Christmas & Happy New Year.   

0 Kudos
Wolfgang_K_
Beginner
1,378 Views

Thank you very much and a Happy New Year! We also had vacation and are back in the office now.

It is good news that the problem can be reproduced at your side. Are there any updates on the state of development so far?

0 Kudos
Peter_W_Intel
Employee
1,378 Views

The developer still is on long holiday, I will keep you update if any news. I will check this issue's state at next week.

0 Kudos
Wolfgang_K_
Beginner
1,378 Views

Hello Peter,
do you have any news from the developer? 
Is it possible to provide a time frame for the fix or at least for the completion of the anaylsis as we cannot use our VTune license until a solution is available?

Thanks

0 Kudos
Peter_W_Intel
Employee
1,378 Views

The good news is that the developer plans to fix this bug, it might be ready in future releases. (Wolfgang K, please send me in private message to let me know about your email box which was registered in product, I can extend your VTune license use for more time)

Another tip (workaround) is that you can profile application with "--profiling-signal" option, This option changes profiling signal the tool use, our developer tested it successfully with values from 33-44, and it should work for signal from SIGRTMIN to SIGRTMAX. 

0 Kudos
Peter_W_Intel
Employee
1,378 Views

Here is an example, based on your example -

amplxe-runss --data-limit-mb=500 --stack-stitching --follow-child --itt-config=frame --stackwalk=offline -profiling-signal 33 -duration 5 --type=cpu:counters:nostack --type=cpu:stack --target-process SignalsTest

0 Kudos
Wolfgang_K_
Beginner
1,378 Views

It is good news that you are planning to fix this bug.

Thanks for the workaround. Unfortunately the SignalsTest application still crashes in our environment (CentOS 5.11, 32 bit) using the command provided. Are there any logfiles or similiar which may help you to analyse why SignalsTest is still crashing on our side? Can we try to use other parameters?

I forwarded your offer concerning extension of the license to our management. They will come back to you next week.

0 Kudos
Peter_W_Intel
Employee
1,378 Views

Maybe it was due to environment or 32bit?

The workaround worked on my side, no crash found - I'm using 2015 U1, Red Hat Enterprise Linux Server release 6.2 (Santiago), 2.6.32-220.el6.x86_64

# amplxe-runss --data-limit-mb=500 --stack-stitching --follow-child --itt-config=frame --stackwalk=offline -profiling-signal 33 -duration 5 --type=cpu:counters:nostack --type=cpu:stack --target-process SignalsTest

amplxe: Collection started
amplxe: Detached
amplxe: Collection stopped - application return code is 0

# amplxe-cl -R hotspots
amplxe: Using result path `/home/peter/problem_report/SignalsTest/src/r000'
amplxe: Executing actions 16 % Resolving information for `libc-2.12.so'        
amplxe: Warning: Cannot locate debugging symbols for file `/lib64/libc-2.12.so'.
amplxe: Executing actions 50 % Generating a report                             
Function  Module       CPU Time:Self  CPU Time:Idle:Self  CPU Time:Poor:Self  CPU Time:Ok:Self  CPU Time:Ideal:Self  CPU Time:Over:Self  Spin Time:Self  Overhead Time:Self
--------  -----------  -------------  ------------------  ------------------  ----------------  -------------------  ------------------  --------------  ------------------
main      SignalsTest          4.990                   0               4.990                 0                    0                   0               0                   0
amplxe: Executing actions 100 % done    

 

I have forwarded your message to developer. Thank you.

0 Kudos
Peter_W_Intel
Employee
1,378 Views

Our developer asked to do:

  1.        export AMPLXE_LOG_LEVEL=TRACE
  2.        reproduce the scenario
  3.        send me collected result folder and related logs from /tmp/amplxe-log-<username> (last 3 folders)

Thank you.

0 Kudos
Wolfgang_K_
Beginner
1,378 Views

Thanks for the input.
Here are the results of the requested tests. Please keep in mind that only the most recent logs were generated with AMPLXE_LOG_LEVEL=TRACE.

# SignalsTest
Starting
Process 11686 attached - interrupt to quit
Process 11686 detached
# When the SignalsTest application is sent Signal 2 now, the process is ended as signal handlers seem to be broken.

# export AMPLXE_LOG_LEVEL=TRACE
# ./amplxe-runss --data-limit-mb=500 --stack-stitching --follow-child --itt-config=frame --stackwalk=offline -profiling-signal 33 -duration 5 --type=cpu:counters:nostack --type=cpu:stack --target-process SignalsTest
amplxe: Collection started
amplxe: Detached
amplxe: Collection stopped - application return code is 0

Our idea was to use VTune from Windows and do remote profiling of our application running under CentOS 5.11 (therefore and becuase we are using VTune with Windows applications as well we acquired a Windows license). 
As the additional parameters can only be given to the command line tool we try to use this to collect the profiling data and use VTune GUI under Windows for the analysis of the collected results.
The following output is therefore just for reference:

# ./amplxe-cl -R hotspots
amplxe: Using result path `/game02/intel/vtune_amplifier_xe_2015.1.1.380310/bin32/r008'
amplxe: Executing actions 14 % Loading data files
amplxe: Error: Cannot load data file `/game02/intel/vtune_amplifier_xe_2015.1.1.380310/bin32/r008/data.0/11660-11686.0-11686.0.strace' ().
amplxe: Executing actions 16 % Resolving information for `libc-2.5.so'
amplxe: Warning: Cannot locate debugging symbols for file `/lib/libc-2.5.so'.
amplxe: Executing actions 50 % done
amplxe: Error: 0x4000001f (No valid license) -- Failed to generate report `hotspots': no valid license can be found (License file is not found. Make sure that your license file is in the correct location and readable.
Tip: Consider setting the license file location with the INTEL_LICENSE_FILE environment variable.).

When opening the results in VTune GUI under Windows 7 (64 bit) we see the following output:

Finalization completed with warnings 
    02.02.2015 12:10:09  Result finalization has completed with warnings that may affect the representation of the analysis data. Please see details below. 
    Cannot load data file `C:\Users\martin\Desktop\Vtune Test 1\r008\data.0\11660-11686.0-11686.0.strace' ().
    Cannot locate file `/game02/bin/SignalsTest'.
    Cannot locate file `/lib/libc-2.5.so'.


I hope this helps us in finding the cause of the problem.

0 Kudos
Oleg_F_Intel
Employee
1,378 Views

Hi Stephan,

Thanks for the logs. As I understand, your problem is in the incorrect ctrl-c handling after collection stop, am I right?

Regards, Oleg

0 Kudos
Wolfgang_K_
Beginner
1,378 Views

Hi Oleg,

Yes that's right, although this is not the only broken handler. For example our real application also uses signal SIGRTMIN+4 which is also broken after collection. Therefore I guess that all signal handlers are broken after collection not just the one for signal 2.

Hope that helps.

Regards,

Stephan

0 Kudos
Oleg_F_Intel
Employee
1,378 Views

Hi Stephan,

I have one more question. I have read your message carefully one more time and i found that you have successfully collect the data with -profiling-signal 33 option. Am I right? Do you have any other blocking issue?

Btw, fix with returning ctrl-c handler will be available in Vtune AXE 2015 U3. 

Regards, Oleg

0 Kudos
Wolfgang_K_
Beginner
1,295 Views

Hi Oleg,

it is good news that a fix will be available in the next release.

Yes, collecting the data does not seem to be the issue for us. But as we need to test our system under load (which we cannot simulate) we still cannot work with VTune until the fix is available. Do you have a time frame for U3?

Regards, Stephan

0 Kudos
Reply