Error while inspecting an executable with itcpin.

[dare2be]

Hello,

as a part of my work assignment, I am currently trying to examine a program. I have made a successful attempt to trace all the MPI calls using your Intel Trace Collector and Analyzer.

However, I would like to take advantage of the ability to trace the User Code which, according to the Intel Trace Collector Reference Guide, is possible through the itcpin utility. However, running the program through itcpin fails. Below is an example output of an attempt to run the program. Trying to insert the VT library (instead of VTfs) produces the same output.

[bash]##### [14:18:03] # [dare2be@schroedinger:~/Praca/IPJ/SuperBayes-v1.5.zMPI.ITAC] #####
$ mpirun -np 4 itcpin --run --profile on --mpi /usr/lib64/mpich2/lib/libmpich.so --insert libVTfs -- ./superbayes_multinest SampleIniFile.ini
E:Function RTN_FindByName called without holding lock. Call PIN_LockClient()/PIN_UnlockClientE:Function RTN_FindByName called without holding lock. Call PIN_LockClient()/PIN_UnlockClientE:Function RTN_FindByName called without holding lock. Call PIN_LockClient()/PIN_UnlockClientE:Function RTN_FindByName called without holding lock. Call PIN_LockClient()/PIN_UnlockClient[/bash]

I can see that the program itself IS running (as shown by the top utility), but after some time, it fails.

The binary is compiled with Intel FORTRAN and C compilers, version 12.0.0

I would be grateful if someone could provide help on this one. If there is a need for more detailed information, I will be more than happy to provide it.

[Dmitry_K_Intel2]

What version of the Intel Trace Analyzer and Collector do you use?
Can you send the output of 'ldd superbayes_multinest"?

I'm not sure that we can guarantee correct work with MPICH2 library but the error "RTN_FindByNamecalledwithoutholdinglock" should not appear anyway.

Can you compile superbayes_multinest with Intel MPI library and run itcpin without '--mpi ...' option?

Regards!
Dmitry

[dare2be]

Thank you for your answer!

The version of the ITAC is 8.0 Update 1. If it's of any help:

[bash]##### [15:27:29] # [dare2be@schroedinger:~] #####
$ ls /opt/intel/itac/
8.0.1.009[/bash]

The output of ldd coming your way...now:

[bash]##### [15:29:18] # [dare2be@schroedinger:~/Praca/IPJ/SuperBayes-v1.5.zMPI.ITAC] #####
$ ldd superbayes_multinest 
	linux-vdso.so.1 =>  (0x00007fff6e3ff000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003853000000)
	libmkl_intel_lp64.so => /opt/intel/composerxe-2011.0.084/mkl/lib/intel64/libmkl_intel_lp64.so (0x00007fe3a93d9000)
	libmkl_sequential.so => /opt/intel/composerxe-2011.0.084/mkl/lib/intel64/libmkl_sequential.so (0x00007fe3a8de9000)
	libmkl_core.so => /opt/intel/composerxe-2011.0.084/mkl/lib/intel64/libmkl_core.so (0x00007fe3a7f49000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003847c00000)
	/home/dare2be/Praca/IPJ/SuperBayes-v1.5.zMPI.ITAC/source/../micromegas/CalcHEP/alpha_s.so (0x00007fe3a7d45000)
	/home/dare2be/Praca/IPJ/SuperBayes-v1.5.zMPI.ITAC/source/../micromegas/CalcHEP/sqme_aux.so (0x00007fe3a7b43000)
	libmpichf90.so.1.2 => /usr/lib64/mpich2/lib/libmpichf90.so.1.2 (0x00007fe3a7940000)
	libmpich.so.1.2 => /usr/lib64/mpich2/lib/libmpich.so.1.2 (0x00007fe3a756e000)
	librt.so.1 => /lib64/librt.so.1 (0x0000003848400000)
	libifport.so.5 => /opt/intel/lib/intel64/libifport.so.5 (0x00007fe3a7434000)
	libifcore.so.5 => /opt/intel/lib/intel64/libifcore.so.5 (0x00007fe3a71d3000)
	libimf.so => /opt/intel/lib/intel64/libimf.so (0x00007fe3a6df5000)
	libsvml.so => /opt/intel/lib/intel64/libsvml.so (0x00007fe3a67bb000)
	libm.so.6 => /lib64/libm.so.6 (0x0000003848000000)
	libintlc.so.5 => /opt/intel/lib/intel64/libintlc.so.5 (0x00007fe3a666c000)
	libc.so.6 => /lib64/libc.so.6 (0x0000003847800000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003852400000)
	libdl.so.2 => /lib64/libdl.so.2 (0x0000003847400000)
	/lib64/ld-linux-x86-64.so.2 (0x0000003847000000)
[/bash]

Right now I am recompiling the project with the Intel MPI library. As soon as I get it to run, I will post the results.

[dare2be]

Hello again.

I just recompiled it with Intel MPI library and it worked...well, sort of. I can't inspect the User Code any deeper than before, it's still only "User Code", I can't see the function call history. Below's the output. Can you make anything sensible out of this?

[bash]##### [16:18:00] # [dare2be@schroedinger:~/Praca/IPJ/SuperBayes-v1.5.zMPI.ITAC] #####
$ mpirun -np 4 itcpin --run --profile --insert libVT -- ./superbayes_multinest SampleIniFile.ini  
(cut)
[0] Intel Trace Collector INFO: Writing tracefile superbayes_multinest.stf in /home/dare2be/Praca/IPJ/SuperBayes-v1.5.zMPI.ITAC
A:Source/pin/pin/image.cpp:ProcessImageUnload:1173: assertion failed: si != rtn_map.end()

################################################################################
## STACK TRACE
################################################################################
addr2line -C -f -e "/opt/intel/itac/8.0.1.009/bin/itcpin32.bin" 0x7faaed8cd180 0x7faaed8cdf41 0x7faaed92fa05 0x7faaed92fdb5 0x7faaed948d56 0x7faaed94b4a0 0x7faaeee6c9ad 0x7faaeef0245a 0x7faaeef47cb5 0x7faaeef4ce6d 0x7faaeee9e8f8 0x7faaeee83bbd 0x7faaeee6b18d 0x7faaeeeff803
A:Source/pin/pin/image.cpp:ProcessImageUnload:1173: assertion failed: si != rtn_map.end()

################################################################################
## STACK TRACE
################################################################################
addr2line -C -f -e "/opt/intel/itac/8.0.1.009/bin/itcpin32.bin" 0x7ffc16156180 0x7ffc16156f41 0x7ffc161b8a05 0x7ffc161b8db5 0x7ffc161d1d56 0x7ffc161d44a0 0x7ffc176f59ad 0x7ffc1778b45a 0x7ffc177d0cb5 0x7ffc177d5e6d 0x7ffc177278f8 0x7ffc1770cbbd 0x7ffc176f418d 0x7ffc17788803
A:Source/pin/pin/image.cpp:ProcessImageUnload:1173: assertion failed: si != rtn_map.end()

################################################################################
## STACK TRACE
################################################################################
addr2line -C -f -e "/opt/intel/itac/8.0.1.009/bin/itcpin32.bin" 0x7f34d8ddb180 0x7f34d8ddbf41 0x7f34d8e3da05 0x7f34d8e3ddb5 0x7f34d8e56d56 0x7f34d8e594a0 0x7f34da37a9ad 0x7f34da41045a 0x7f34da455cb5 0x7f34da45ae6d 0x7f34da3ac8f8 0x7f34da391bbd 0x7f34da37918d 0x7f34da40d803
A:Source/pin/pin/image.cpp:ProcessImageUnload:1173: assertion failed: si != rtn_map.end()

################################################################################
## STACK TRACE
################################################################################
addr2line -C -f -e "/opt/intel/itac/8.0.1.009/bin/itcpin32.bin" 0x7fd3650fb180 0x7fd3650fbf41 0x7fd36515da05 0x7fd36515ddb5 0x7fd365176d56 0x7fd3651794a0 0x7fd36669a9ad 0x7fd36673045a 0x7fd366775cb5 0x7fd36677ae6d 0x7fd3666cc8f8 0x7fd3666b1bbd 0x7fd36669918d 0x7fd36672d803
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
Pin 2.7
Copyright (c) 2003-2010, Intel Corporation. All rights reserved.
@CHARM-VERSION: $Id: version.cpp 35789 2010-08-04 21:15:41Z tevi $
@CHARM-BUILDER: BUILDER
@CHARM-COMPILER: gcc 3.4.6
@CHARM-TARGET: ia32e
@CHARM-CFLAGS:  __OPTIMIZE__=1  __NO_INLINE__=__NO_INLINE__
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
Pin 2.7
Copyright (c) 2003-2010, Intel Corporation. All rights reserved.
@CHARM-VERSION: $Id: version.cpp 35789 2010-08-04 21:15:41Z tevi $
@CHARM-BUILDER: BUILDER
@CHARM-COMPILER: gcc 3.4.6
@CHARM-TARGET: ia32e
@CHARM-CFLAGS:  __OPTIMIZE__=1  __NO_INLINE__=__NO_INLINE__
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
Pin 2.7
Copyright (c) 2003-2010, Intel Corporation. All rights reserved.
@CHARM-VERSION: $Id: version.cpp 35789 2010-08-04 21:15:41Z tevi $
@CHARM-BUILDER: BUILDER
@CHARM-COMPILER: gcc 3.4.6
@CHARM-TARGET: ia32e
@CHARM-CFLAGS:  __OPTIMIZE__=1  __NO_INLINE__=__NO_INLINE__
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
??
??:0
Pin 2.7
Copyright (c) 2003-2010, Intel Corporation. All rights reserved.
@CHARM-VERSION: $Id: version.cpp 35789 2010-08-04 21:15:41Z tevi $
@CHARM-BUILDER: BUILDER
@CHARM-COMPILER: gcc 3.4.6
@CHARM-TARGET: ia32e
@CHARM-CFLAGS:  __OPTIMIZE__=1  __NO_INLINE__=__NO_INLINE__

[/bash]

EDIT: I noticed that itcpin tries to run /opt/intel/itac/8.0.1.009/bin/itcpin32.bin (see above output). When I tried to run /opt/intel/itac/8.0.1.009/bin/itcpin64.bin by hand, I got memory access violation error (or whatever polish "naruszenie ochrony pamici" equals to in english). How do I get itcpin to run the 64-bit version?

[Dmitry_K_Intel2 · Accepted solution]

PIN can check whether the code is 32-bit or 64-bit and invoke appropriate itcpin{arch}.bin
Might be you code calls something which is 32-bit (might be some other program).

It seems to me that we see a conflict in the VT library but it doesn't matter.

Assuming that you have source code I'd suggest you using of '-tcollect' option to the Intel compiler. This option will instrument all user functions and after that you just start the application like 'mpirun -np 4 ./your_app' and a trace file will be created.
After that investigate it in Trace Analyzer. In the Trace Analyzer in Function Profile click the right button on the "Group Application" and choose "Ungroup Application", after that right click on a source file and press "Ungroup /name/of/source/file.c".
Functions from that source file will be seen.

I hope this solves your problem.

Regards!
Dmitry

[dare2be]

Thank you for your fast reply.

Yes, I do have acces to the source code. However, I'd rather not fiddle with it too much (it's close to 100k lines of undocumented code, not to mention self-written bash-based autotools which have been giving me a headache for the last week). But, following your advice, I recompiled it with '-tcollect'. It almost worked.

When it came to linking, linker complained about the VT library not being position-independent ('relocation R_X86_64_32 against `_2__STRING.7 ' can not be used when making a shared object; recompile with -fPIC'). You can see in the ldd output above that the source code creates two shared libraries, against which the executable is later linked.

Is there any position-independent version of VT available? I'd appreciate it - I feel I'm pretty close to getting a complete trace :)

Sincerely,
Wojciech

[Dmitry_K_Intel2]

Hi Wojciech,

This happens because '-tcollect' option link static version of the LibVT with an application.
To use dynamic dynamic version instead you may add a parameter to the '-tcollect' option.
$ mpiicc -tcollect /opt/intel/itac/8.0.1.009/slib_impi4/libVT.so ...your_options...

I hope that everything will be fine in this case.

Regards!
Dmitry

[dare2be]

Yes! thank you.

There was another problem - after few minutes, I had about 20 GB of trace data. But I'll be able to sort it out using filters (hopefully).

Once again, thank you for your assistance.

Wojtek

[Dmitry_K_Intel2]

Wojtek,

'-tcollect-filter' may help you to reduce the size of a trace file, but unfortunately it was implemented so that even if you filter out a user function, MPI functions called from this function will exist in the trace.

As soos as you have source files you can use ITAC API - you can use vt_traceon()/vt_traceoff() functions to enable and disable tracing respectively.

Regards!
Dmitry