VTune hangs when launched as a specific user

[Mateusz_W_]

Hi,

We're experiencing an issue with where VTune hangs if run as a specific user. Please note - this user is capable of running VTune i.e. by attaching it to a running process, but NOT if run as following:

/usr/local/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect snb-general-exploration -- ls &

This results in a hang:

[1] 12246
% ps aux | grep <username>
username  12246  0.8  0.0 230216 33656 pts/1    SNl  19:11   0:00 /usr/local/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect snb-general-exploration -- ls
username  12248  0.0  0.0 230216 12760 pts/1    SN   19:11   0:00 /usr/local/intel/vtune_amplifier_xe_2013/bin64/amplxe-cl -collect snb-general-exploration -- ls
 

% gdb -p 12248

(gdb) bt
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007ffff51e9267 in void boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >(boost::unique_lock<boost::mutex>&) ()
   from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_qfagentminidump_core_1.12.so
#2  0x00007ffff51e4fcc in qfagent_1::exceptionSignalHandler(int, siginfo*, void*) ()
   from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_qfagentminidump_core_1.12.so
#3  <signal handler called>
#4  0x00007ffff1eee1b5 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#5  0x00007ffff1ef0fc0 in *__GI_abort () at abort.c:92
#6  0x00007ffff34df013 in featurestat_1_0_22::ism_accessor_impl_t::invoke_ism(featurestat_1_0_22::ism_accessor_impl_t::ism_tool_t, std::vector<std::string, std::allocator<std::string> > const&, bool) const () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#7  0x00007ffff34e187d in featurestat_1_0_22::ism_accessor_impl_t::is_feedback_enabled() const ()
   from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#8  0x00007ffff34ce254 in featurestat_1_0_22::statistic_impl_t::is_enabled() const ()
   from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#9  0x00007ffff34ccc17 in featurestat_1_0_22::statistic_impl_t::flush() () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#10 0x00007ffff7ba7b18 in cctrl2::CliManager::CliManager() () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_cctrl_climgr_2.31.so
#11 0x00007ffff7ba7889 in cctrl2::ICliManager::create() () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_cctrl_climgr_2.31.so
#12 0x0000000000409867 in ?? ()
#13 0x0000000000409178 in ?? ()
#14 0x00007ffff1edac8d in __libc_start_main (main=<optimized out>, argc=<optimized out>, ubp_av=<optimized out>, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffeb38) at libc-start.c:228

% gdb -p 12246

(gdb) bt
#0  0x00007ffff4b2dbdd in __libc_waitpid (pid=12248, stat_loc=<optimized out>, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:41
#1  0x00007ffff34df181 in featurestat_1_0_22::ism_accessor_impl_t::invoke_ism(featurestat_1_0_22::ism_accessor_impl_t::ism_tool_t, std::vector<std::string, std::allocator<std::string> > const&, bool) const () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#2  0x00007ffff34e187d in featurestat_1_0_22::ism_accessor_impl_t::is_feedback_enabled() const ()
   from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#3  0x00007ffff34ce254 in featurestat_1_0_22::statistic_impl_t::is_enabled() const ()
   from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#4  0x00007ffff34ccc17 in featurestat_1_0_22::statistic_impl_t::flush() () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so
#5  0x00007ffff7ba7b18 in cctrl2::CliManager::CliManager() () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_cctrl_climgr_2.31.so
#6  0x00007ffff7ba7889 in cctrl2::ICliManager::create() () from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_cctrl_climgr_2.31.so
#7  0x0000000000409867 in ?? ()
#8  0x0000000000409178 in ?? ()
#9  0x00007ffff1edac8d in __libc_start_main (main=<optimized out>, argc=<optimized out>, ubp_av=<optimized out>, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fffffffeb38) at libc-start.c:228

Killing the child process (12248 above) resolves the hang and VTune runs as one would expect.

Thanks!

Mateusz

[Bernard]

Could the hang be related to synchronization issue like deadlock?

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
#1  0x00007ffff51e9267 in void boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >(boost::unique_lock<boost::mutex>&) ()

[Mateusz_W_]

Yes. Please note the stacks I copied are in the context of the VTune Amplifier process.

[Mateusz_W_]

We have found a solution to this problem: this seems to occur only when VTune does not have read permissions to the $HOME directory of the user who called it via sudo -Eu. 

[Peter_W_Intel]

Mateusz W. wrote:

We have found a solution to this problem: this seems to occur only when VTune does not have read permissions to the $HOME directory of the user who called it via sudo -Eu. 

Usually if experienced such problem, we need to verify as below steps:

1. Make sure that vtune drivers are loaded, for example - "lsmod | grep sep3", "lsmod | grep pax", and "lsmod | grep vtsspp"

2. Check if your account is in vtune group, default "vtune", "vim /etc/group" 

3. Check if your application can work without VTune

4. Use "amplxe-cl -collect snb-general-exploration -r /tmp/r000ge -- /bin/ls" to know if it is write-permission issue.

5. Finally analyze your real application by using VTune.

[Bernard]

>>>Please note the stacks I copied are in the context of the VTune Amplifier process.>>>

Yes I noticed that.

There are also two function calls one of them is called call from /usr/local/intel/vtune_amplifier_xe_2013/bin64/../lib64/libamplxe_featurestat_core_1.0.so

which may indicate code abortion like circumstances.