Intel Thread Checker 3.0 for Linux deadlocks during startup

[bartoschek]

Hi,

I am currently evaluating the Intel Thread Checker for our application. Small testcases like the supplied program from the getting started guide work fine, but our main application does not start. Here is the output of a run with the thread checker:

-bash-3.00$ tcheck_cl -p verbose prog
Intel Thread Checker 3.0 command line instrumentation driver (23479)
Copyright (c) 2006 Intel Corporation. All rights reserved.
Building project
Instrumenting
12% prog ( All Functions ):.......................................................
.
37% libc-2.3.4.so ( Minimal ):....
50% libdl-2.3.4.so ( Minimal ):..
62% libgcc_s.so.1 ( Minimal ):..
75% libm-2.3.4.so ( Minimal ):..
87% libpthread-2.3.4.so ( Minimal ):..
100% libstdc++.so.6.0.7 ( Minimal ):..

Running: prog

[Intel Thread Checker for OpenMP* and POSIX* Threads 4.9.12]
[Copyright 1997-2006 by Intel Corporation]
[Intel Thread Checker Information: For usage information use TC_OPTIONS help]
[Intel Thread Checker Options: verbose,terminate,file=threadchecker.thr]
[Intel Thread Checker Status: Memory 1.0 MB allocated, 0.00x increase, 0 routines]
[ Memory usage(total): 585.1 KB ]
[ Memory usage(map): 72.5 KB ]
[ Memory usage(vector): 512.0 KB ]
[ Memory usage(mem_a): 0.0 KB ]
[ Memory usage(mem_b): 0.0 KB ]
[ Memory usage(module): 0.0 KB ]
[ Memory usage(graph): 0.0 KB ]
[ Memory usage(dead): 0.0 KB ]
[ Memory usage(thr): 0.0 KB ]
[ Memory usage(misc): 0.5 KB ]
[Intel Thread Checker Status: Thread #1 created]
[Intel Thread Checker Information: Thu Mar 8 16:05:30 2007 - Initialized]

I see four processes started including my own application. When I attach to it, it has two threads started. Both threads are waiting for a mutex and have the same traceback:

#0 0x00000035f4b0ae2b in __lll_mutex_lock_wait () from /tmp/bartosch_tc_cl_cache/libpthread.so.0
#1 0x00007fffffffe4d8 in ?? ()
#2 0x00007fffffffe401 in ?? ()
#3 0x00000035f4b07bba in ?? () from /tmp/bartosch_tc_cl_cache/libpthread.so.0
#4 0x0000000000000008 in ?? ()
#5 0x00002aaaaab04657 in __kmd_allocate () from /opt/intel/itt/tcheck/bin/32e/libassuret40.so
#6 0x00002aaaaab059d1 in __kmd_func_node_create () from /opt/intel/itt/tcheck/bin/32e/libassuret40.so
#7 0x00002aaaaab062c9 in __kmd_push_branch () from /opt/intel/itt/tcheck/bin/32e/libassuret40.so
#8 0x00002aaaaaad2d4b in __kmd_impl_enter_function_call () from /opt/intel/itt/tcheck/bin/32e/libassuret40.so
#9 0x00002aaaaaad5bba in __kmd_aux_run () from /opt/intel/itt/tcheck/bin/32e/libassuret40.so
#10 0x00002aaaaaadaa8d in __kmd_run_flush () from /opt/intel/itt/tcheck/bin/32e/libassuret40.so
#11 0x00002aaaaaade42f in __kmdc_check_call_stack () from /opt/intel/itt/tcheck/bin/32e/libassuret40.so
#12 0x00002aaaaac8e8db in CheckStack () from /tmp/bartosch_tc_cl_cache/libtctrace.so
#13 0x0000000001a34f7a in ?? ()
#14 0x0000000000000000 in ?? ()

There are also two processes BCDriver_TC32e waiting in waitpid().


Because the deadlock occurs during the startup and in code from Intel Thread Checker, I think that this deadlock is caused internally by Intel Thread Checker.

How to get rid of it?

[TimP]

We have been trying (unsucessfully, so far)to get access to this version, so my experience is with the prior internal version.

Generally, dynamic instrumentation is not supported for large applications, as it has excessive resource requirements.So, the entire application (at least all threaded portions) must be rebuilt with -tcheck option, which sets -O0 -g. So, the application must be capable of running with those options, ruling out large IA-64 applications. In particular, I have had to replace all threaded libraries, such as Intel MKL, with equivalent source code.