- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
we are using Intel VTune 2015 for profiling our application which is running under CentOS 5.11.
Our application uses c++ signals for the control flow. When trying to do a basic hotspots analysis using amplxe-cl command line tool with the following parameters:
-duration 20 --run-pass-thru=--profiling-signal=1
VTune yields the following error message when detaching after the 20 seconds duration. Alternative to the number 1 I also tried number 4 without any change in results.
amplxe: Error: Assertion failed: handler_ex1445: obj->is_first_class_handler_set[signo] == 1 : BUG! : signo == 1. Please contact the technical support.
Without using "--run-pass-thru=--profiling-signal=1" as parameter the signal handlers of our application do not work after profiling and the process is ended when receiving a signal.
Please provide guidance on how to use VTune in this scenario.
Best Regards,
Stephan
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
As default, VTune Amplifier will use application's alter signal stack which is 64KB limited. So, you may not use VTune's alter signal stack from application's. For example:
amplxe-cl –collect hotspots –run-pass-thru=-no-altstack -- application
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Peter,
thanks for your response. We have been trying the solution you have proposed.
Unfortunately the result for us does not change. When our application receives a signal after VTune is detached the process is just shut down without our handlers taking care of the signal.
Please keep in mind that we do not set up an alternate signal stack using function sigaltstack.
We do the following to set up the signal handlers on our side.
- Add signal handlers using sigaction
We use sigfillset on our sigactions to ignore all other signals while a handler is running
The sa_flags are set to SA_RESTART
- For SIGINT, SIGQUIT, SIGTERM, SIGSEGV, SIGFPE, SIGILL, SIGABRT we set handler functions
- SIGHUP, SIGCHLD, SIGPIPE, SIGALRM, SIGXFSZ, SIGSTKFLT, SIGTSTP, SIGTTIN, SIGTTOU, SIGURG, SIGXCPU, SIGVTALRM, SIGWINCH, SIGPWR, SIGIO we set to SIG_ING
- We use pthread_sigmask(SIG_SETMASK, &SignalSet, NULL) with
sigset_t SignalSet;
sigfillset(&SignalSet);
sigdelset(&SignalSet, SIGINT);
sigdelset(&SignalSet, SIGQUIT);
sigdelset(&SignalSet, SIGTERM);
sigdelset(&SignalSet, SIGUSR1);
sigdelset(&SignalSet, SIGUSR2);
to block all signals except for the ones needed.
- When receiving a signal other that SIGUSR2 in our handler, we set the handler for the above signals to SIG_IGN to ignore further signals
Therefor we were trying to use "--run-pass-thru=--profiling-signal" as we though this would tell VTune to use a signal which is not used on our side, hoping to preserve our signal handlers.
Could you please provide further details on the usage of signals with VTune and how to handle the issue on our side?
Best Regards,
Stephan
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Is it possible that you can provide vtune's result directory for investigating?
I have escalated this issue to the developer, please feel free to send me private message with attached vtune result. Thank you.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I got comment from developer - "Could you please ask the customer to try to pass any unused signal from SIGRTMIN..SIGRTMAX range to --profiled signal option?"
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the input. We tried the --profiling-signal option using a signal between SIGRTMIN..SIGRTMAX with no improvement.
We managed to implement a small testapplication having the same signalhandling as our real application. Please find it attached to this post. Maybe a developer on your side can use it to reproduce the error and provide input on how to fix the issue.
Please perform the following steps for reproduction:
extract: tar -zxf SignalsTest.tar.gz
go to SignalsTest/Debug
run: make all
run:
./SignalsTest
kill -12 <PID of SignalsTest>
kill -2 <PID of SignalsTest>
kill -9 <PID of SignalsTest>
result:
Starting
Caught 12.
Caught 2.
Killed
run:
./SignalsTest
kill -12 <PID of SignalsTest>
./amplxe-cl -collect hotspots -duration 10 -target-process SignalsTest
kill -2 <PID of SignalsTest>
result:
Starting
Caught 12.
Process is killed when receiving signal 2 as handler is broken after VTune analysis.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for you file, I have forwarded this to the developer.
It is vacation season for now - the response might be a little bit slow. Thanks for your understandings:-)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Just let you know, the developer confirmed that this problem can be reproducible...it was under investigating. The solution will be slow due to Holiday. Merry Christmas & Happy New Year.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thank you very much and a Happy New Year! We also had vacation and are back in the office now.
It is good news that the problem can be reproduced at your side. Are there any updates on the state of development so far?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The developer still is on long holiday, I will keep you update if any news. I will check this issue's state at next week.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello Peter,
do you have any news from the developer?
Is it possible to provide a time frame for the fix or at least for the completion of the anaylsis as we cannot use our VTune license until a solution is available?
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The good news is that the developer plans to fix this bug, it might be ready in future releases. (Wolfgang K, please send me in private message to let me know about your email box which was registered in product, I can extend your VTune license use for more time)
Another tip (workaround) is that you can profile application with "--profiling-signal" option, This option changes profiling signal the tool use, our developer tested it successfully with values from 33-44, and it should work for signal from SIGRTMIN to SIGRTMAX.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here is an example, based on your example -
amplxe-runss --data-limit-mb=500 --stack-stitching --follow-child --itt-config=frame --stackwalk=offline -profiling-signal 33 -duration 5 --type=cpu:counters:nostack --type=cpu:stack --target-process SignalsTest
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It is good news that you are planning to fix this bug.
Thanks for the workaround. Unfortunately the SignalsTest application still crashes in our environment (CentOS 5.11, 32 bit) using the command provided. Are there any logfiles or similiar which may help you to analyse why SignalsTest is still crashing on our side? Can we try to use other parameters?
I forwarded your offer concerning extension of the license to our management. They will come back to you next week.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Maybe it was due to environment or 32bit?
The workaround worked on my side, no crash found - I'm using 2015 U1, Red Hat Enterprise Linux Server release 6.2 (Santiago), 2.6.32-220.el6.x86_64
# amplxe-runss --data-limit-mb=500 --stack-stitching --follow-child --itt-config=frame --stackwalk=offline -profiling-signal 33 -duration 5 --type=cpu:counters:nostack --type=cpu:stack --target-process SignalsTest
amplxe: Collection started
amplxe: Detached
amplxe: Collection stopped - application return code is 0
# amplxe-cl -R hotspots
amplxe: Using result path `/home/peter/problem_report/SignalsTest/src/r000'
amplxe: Executing actions 16 % Resolving information for `libc-2.12.so'
amplxe: Warning: Cannot locate debugging symbols for file `/lib64/libc-2.12.so'.
amplxe: Executing actions 50 % Generating a report
Function Module CPU Time:Self CPU Time:Idle:Self CPU Time:Poor:Self CPU Time:Ok:Self CPU Time:Ideal:Self CPU Time:Over:Self Spin Time:Self Overhead Time:Self
-------- ----------- ------------- ------------------ ------------------ ---------------- ------------------- ------------------ -------------- ------------------
main SignalsTest 4.990 0 4.990 0 0 0 0 0
amplxe: Executing actions 100 % done
I have forwarded your message to developer. Thank you.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Our developer asked to do:
- export AMPLXE_LOG_LEVEL=TRACE
- reproduce the scenario
- send me collected result folder and related logs from /tmp/amplxe-log-<username> (last 3 folders)
Thank you.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the input.
Here are the results of the requested tests. Please keep in mind that only the most recent logs were generated with AMPLXE_LOG_LEVEL=TRACE.
# SignalsTest
Starting
Process 11686 attached - interrupt to quit
Process 11686 detached
# When the SignalsTest application is sent Signal 2 now, the process is ended as signal handlers seem to be broken.
# export AMPLXE_LOG_LEVEL=TRACE
# ./amplxe-runss --data-limit-mb=500 --stack-stitching --follow-child --itt-config=frame --stackwalk=offline -profiling-signal 33 -duration 5 --type=cpu:counters:nostack --type=cpu:stack --target-process SignalsTest
amplxe: Collection started
amplxe: Detached
amplxe: Collection stopped - application return code is 0
Our idea was to use VTune from Windows and do remote profiling of our application running under CentOS 5.11 (therefore and becuase we are using VTune with Windows applications as well we acquired a Windows license).
As the additional parameters can only be given to the command line tool we try to use this to collect the profiling data and use VTune GUI under Windows for the analysis of the collected results.
The following output is therefore just for reference:
# ./amplxe-cl -R hotspots
amplxe: Using result path `/game02/intel/vtune_amplifier_xe_2015.1.1.380310/bin32/r008'
amplxe: Executing actions 14 % Loading data files
amplxe: Error: Cannot load data file `/game02/intel/vtune_amplifier_xe_2015.1.1.380310/bin32/r008/data.0/11660-11686.0-11686.0.strace' ().
amplxe: Executing actions 16 % Resolving information for `libc-2.5.so'
amplxe: Warning: Cannot locate debugging symbols for file `/lib/libc-2.5.so'.
amplxe: Executing actions 50 % done
amplxe: Error: 0x4000001f (No valid license) -- Failed to generate report `hotspots': no valid license can be found (License file is not found. Make sure that your license file is in the correct location and readable.
Tip: Consider setting the license file location with the INTEL_LICENSE_FILE environment variable.).
When opening the results in VTune GUI under Windows 7 (64 bit) we see the following output:
Finalization completed with warnings
02.02.2015 12:10:09 Result finalization has completed with warnings that may affect the representation of the analysis data. Please see details below.
Cannot load data file `C:\Users\martin\Desktop\Vtune Test 1\r008\data.0\11660-11686.0-11686.0.strace' ().
Cannot locate file `/game02/bin/SignalsTest'.
Cannot locate file `/lib/libc-2.5.so'.
I hope this helps us in finding the cause of the problem.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Stephan,
Thanks for the logs. As I understand, your problem is in the incorrect ctrl-c handling after collection stop, am I right?
Regards, Oleg
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Oleg,
Yes that's right, although this is not the only broken handler. For example our real application also uses signal SIGRTMIN+4 which is also broken after collection. Therefore I guess that all signal handlers are broken after collection not just the one for signal 2.
Hope that helps.
Regards,
Stephan
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Stephan,
I have one more question. I have read your message carefully one more time and i found that you have successfully collect the data with -profiling-signal 33 option. Am I right? Do you have any other blocking issue?
Btw, fix with returning ctrl-c handler will be available in Vtune AXE 2015 U3.
Regards, Oleg
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Oleg,
it is good news that a fix will be available in the next release.
Yes, collecting the data does not seem to be the issue for us. But as we need to test our system under load (which we cannot simulate) we still cannot work with VTune until the fix is available. Do you have a time frame for U3?
Regards, Stephan

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page