Illegal instruction in libiomp5 when upgrading from ipp 7.0.7

[Francesco_Lamonica]

Hi all,

i am running an IPP based application on a GEODE cpu running a linux (debian etch) 32bit distribution.

With ipp up to 7.0.5 all is fine. I recently upgraded to IPP 7.0.7 and i get an illegal instruction at startup

the backtrace i get is the following:

(gdb) r

Starting program: /opt/agent/testingprog

[Thread debugging using libthread_db enabled]

[New Thread -1223795808 (LWP 12265)]

Program received signal SIGILL, Illegal instruction.

[Switching to Thread -1223795808 (LWP 12265)]

rml::internal::internalPoolMalloc (mPool=0xb76f2540, size=11) at ../../src/tbbmalloc/frontend.cpp:1917

1917 ../../src/tbbmalloc/frontend.cpp: No such file or directory.

in ../../src/tbbmalloc/frontend.cpp

(gdb) where

#0 rml::internal::internalPoolMalloc (mPool=0xb76f2540, size=11) at ../../src/tbbmalloc/frontend.cpp:1917

#1 0xb76d0b6e in scalable_malloc (size=11) at ../../src/tbbmalloc/frontend.cpp:2194

#2 0xb7682ad2 in __kmp_env_get () from /opt/agent/lib/libiomp5.so

#3 0xb768804c in __kmp_i18n_catopen () from /opt/agent/lib/libiomp5.so

#4 0xb7688950 in __kmp_i18n_catgets () from /opt/agent/lib/libiomp5.so

#5 0xb7688852 in __kmp_msg_format () from /opt/agent/lib/libiomp5.so

#6 0xb7690858 in __kmp_rml_report_error_handler () from /opt/agent/lib/libiomp5.so

#7 0xb76b1c4a in __kmp::dynamic_link () from /opt/agent/lib/libiomp5.so

#8 0xb769ac45 in __kmp_do_serial_initialize () from /opt/agent/lib/libiomp5.so

#9 0xb769ab66 in __kmp_middle_initialize () from /opt/agent/lib/libiomp5.so

#10 0xb7684435 in omp_get_num_procs () from /opt/agent/lib/libiomp5.so

#11 0xb762b9d6 in _init () from /opt/agent/lib/libippcore.so.7.0

#12 0xbf826278 in ?? ()

#13 0xb764f4c8 in _init () from /opt/agent/lib/libiomp5.so

#14 0xb7f1d921 in _dl_rtld_di_serinfo () from /lib/ld-linux.so.2

#15 0xb7f127ef in ?? () from /lib/ld-linux.so.2

[Gennady_F_Intel]

this is an unknown issue - both of these versions contain the same versions of threading libraries( libiomp ) if I remember exactly. you can check it by himself. what you were changing in your application after IPP's binaries were updated? Did you change linking line?

[Francesco_Lamonica]

Hi Gennady,

executing a diff between the two libiomp5.so files returns a difference.

The scenario is the following

testapp links against testlib

testlib uses libspeech.a (from ipp-samples) that uses libipps and libippsc functions.

(libspeech was obviously recompiled from 7.0.5 and 7.0.7)

the code (and Makefile) of the testlibrary has not changed

the code (and Makefile) of the testapp has not changed.

besides from what i gather from stack trace this seems to happen before the testlib is being loaded...

let me know if you need further info

[Francesco_Lamonica]

Any news about this topic?

[Gennady_F_Intel]

We cannot reproduce the issue on our side.can you give the reproducible test for us?

[Francesco_Lamonica]

Hi Gennady,

the actual code can be reduced to a simple

#include

int main() {

std::cout << "Hello world" << std::endl;

return 0;

}

this simple code linked against libiomp5, libippcore

with the line: g++ -Wall -g test.cpp -L -liomp5 -lippcore -o test

it was compiled with gcc 4.1.2 (debian etch on geode) and works with ipp 7.0.5, but gives an illegal instruction with ipp 7.0.7

etch:/# ldd ./test-ipp707

linux-gate.so.1 => (0xb7f31000)

libiomp5.so => /root/libiomp5.so (0xb7e73000)

libippcore.so.7.0 => /root/libippcore.so.7.0 (0xb7e5c000)

libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7d73000)

libm.so.6 => /lib/tls/libm.so.6 (0xb7d4e000)

libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7d43000)

libc.so.6 => /lib/tls/libc.so.6 (0xb7c10000)

libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7bfe000)

libdl.so.2 => /lib/tls/libdl.so.2 (0xb7bfa000)

/lib/ld-linux.so.2 (0xb7f32000)

etch:/# ./test-ipp707

Illegal instruction

[Ying_H_Intel]

Hi Francesco,

Seeing fromthe simpletest code, it seems nothing to do with the libiomp5 (openmp library)and ipp libraries. But anyway, could you pleaseshow the gdb resultwhen run the "test-ipp707"?

(in the test code,the"test" is"test-ipp707", right?). and please export theOpenMP version alsoas below.
[U105439illegal]$ export KMP_VERSION=1
[U105439illegal]$ gdb test

Thanks
Ying

[Francesco_Lamonica]

Hello Ying,

here is what you asked for :)

/tmp/ipp-illegal-instruction-test# gdb ./test

GNU gdb 6.4.90-debian

Copyright (C) 2006 Free Software Foundation, Inc.

GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.

Type "show copying" to see the conditions.

There is absolutely no warranty for GDB. Type "show warranty" for details.

This GDB was configured as "i486-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

(gdb) r

Starting program: /rw/tmp/ipp-illegal-instruction-test/test

[Thread debugging using libthread_db enabled]

[New Thread -1211778208 (LWP 11595)]

Program received signal SIGILL, Illegal instruction.

[Switching to Thread -1211778208 (LWP 11595)]

rml::internal::internalPoolMalloc (mPool=0xb7f87540, size=11) at ../../src/tbbmalloc/frontend.cpp:1917

1917 ../../src/tbbmalloc/frontend.cpp: No such file or directory.

in ../../src/tbbmalloc/frontend.cpp

(gdb)

However i do not understand why you say it has nothing to do with iomp5 and ipp... the simple snippet does (almost) nothing and we get the illegal instruction only when is linked against libiomp5 and ipp. The stack trace shows how the calls from the loader are up to the libiomp malloc call (before any actual code is being run)

Or am i reading the stack trace wrong?

thanks

[Ying_H_Intel]

Hi Francesco,

Thansk for the test. i see. I assumed, if without libiomp5 and libippcore, the command: g++ -Wall -g test.cpp -o test should be work fine. sothe codehas nothing to do with libiomp5 and ipp.But the binary did.


Could you pleasetry the command :
1)g++ -Wall -g test.cpp -L -liomp5 -o test
2)g++ -Wall -g test.cpp -L -lippcore -liomp5-o test

andseeifthey can run?

We want to make sure if it is libiomp5 issue or ipp issue orthe combination's order?
and pleaseset environment export KMP_VERSION=1 before run them.

Best Regards,
Ying

[Pavel_B_Intel1]

Hello Francesco,

do you use standalone IPP package or Intel Composer 2011? did you update IPP verion only or Intel Composer too?

Pavel

[Francesco_Lamonica]

Hi Ying,

here is the report of the tests

== On the Host ==

vm-etch-dev32# cat ipp_test.cpp
#include

int main(int argc, char** argv)
{
std::cout << "testing ipp" << std::endl;
return 0;
}

vm-etch-dev32# g++ -Wall -g ipp_test.cpp -L./ -liomp5 -o ipp_test_1

vm-etch-dev32# g++ -Wall -g ipp_test.cpp -L./ -lippcore -liomp5 -o ipp_test_2

vm-etch-dev32# ls
ipp_test ipp_test_1 ipp_test_2 ipp_test.cpp libiomp5.so libippcore.so libippcore.so.7.0 test

== On the Target ==

etch:/tmp/ipp-illegal-instruction-test# ldd ipp_test_1
linux-gate.so.1 => (0xb7f8c000)
libiomp5.so => /tmp/ipp-illegal-instruction-test/libiomp5.so (0xb7ece000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7de5000)
libm.so.6 => /lib/tls/libm.so.6 (0xb7dc0000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7db5000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7c83000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7c70000)
libdl.so.2 => /lib/tls/libdl.so.2 (0xb7c6c000)
/lib/ld-linux.so.2 (0xb7f8d000)

etch:/tmp/ipp-illegal-instruction-test# ldd ipp_test_2
linux-gate.so.1 => (0xb7ef9000)
libippcore.so.7.0 => /tmp/ipp-illegal-instruction-test/libippcore.so.7.0 (0xb7ee0000)
libiomp5.so => /tmp/ipp-illegal-instruction-test/libiomp5.so (0xb7e24000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7d3b000)
libm.so.6 => /lib/tls/libm.so.6 (0xb7d16000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7d0b000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7bd8000)
libdl.so.2 => /lib/tls/libdl.so.2 (0xb7bd4000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7bc2000)
/lib/ld-linux.so.2 (0xb7efa000)

etch:/tmp/ipp-illegal-instruction-test# ./ipp_test_1
testing ipp
etch:/tmp/ipp-illegal-instruction-test# ./ipp_test_2
Illegal instruction

so it seems the problem lies in libippcore

[Francesco_Lamonica]

Hi Pavel,

i am using only IPP, the compiler has always been gcc 4.1.2

[Pavel_B_Intel1]

Francesco,

unfortunatelly I don't have appropriate HW to test your case. Could you help me and build 3 variant of application:
1) build your test application and link with libippcore.so only
2) build your test application and link with libiomp5.so only
3) extend your test application to add:

extern int omp_get_num_procs (void);
.....
main()
{
....
int p= omp_get_num_procs();
....
}

and link it with libiomp5.so only

Thanks, Pavel

[Francesco_Lamonica]

Hi Pavel,

as you can see from my post #10 case 2) was already tested and it printed the correct string

i tried now also case 1) and case 3) and the results are:

case 1: i get the illegal instruction

case 3: i could not test on the geode board since it did not compile

vm-etch-dev32# cat ipp_test_intel_get_procs.cpp
#include

extern int omp_get_num_procs (void);

int main(int argc, char** argv)
{
int p = omp_get_num_procs();
std::cout << "testing ipp: no of procs is" << p << std::endl;
return 0;
}

vm-etch-dev32# g++ -Wall -g ipp_test_intel_get_procs.cpp -L./ -liomp5 -o ipp_test_get_procs
/tmp/cckp3bTU.o: In function `main':
/root/sandbox/ipp-illegal-instruction-test/ipp_test_intel_get_procs.cpp:7: undefined reference to `omp_get_num_procs()'
collect2: ld returned 1 exit status

to sum it up:

vm-etch-dev32# cat ipp_test.cpp
#include

int main(int argc, char** argv)
{
std::cout << "testing ipp" << std::endl;
return 0;
}

vm-etch-dev32# g++ -Wall -g ipp_test.cpp -L./ -liomp5 -o ipp_test_iomp5_only
vm-etch-dev32# g++ -Wall -g ipp_test.cpp -L./ -lippcore -o ipp_test_ippcore_only
vm-etch-dev32# g++ -Wall -g ipp_test.cpp -L./ -lippcore -liomp5 -o ipp_test_ippcore_and_iomp5

etch:/tmp/ipp-illegal-instruction-test# ./ipp_test_iomp5_only
testing ipp
etch:/tmp/ipp-illegal-instruction-test# ./ipp_test_ippcore_only
Illegal instruction
etch:/tmp/ipp-illegal-instruction-test# ./ipp_test_ippcore_and_iomp5
Illegal instruction

the one with the omp_get_num_procs() did not compile

[Pavel_B_Intel1]

Hi Francesco,

sorry to disturb you the right source is:

#include

extern "C" {
extern int omp_get_num_procs (void);
}

int main(int argc, char** argv)
{
int p = omp_get_num_procs();
std::cout << "testing ipp: no of procs is" << p << std::endl;
return 0;
}

g++ test.cpp -o test -L /nfs/site/proj/ipp/build/ipp/ipp70gold-20120406-7-0-0205-105/comp_lib/linux32 -liomp5 -lpthread

one more question: why you don't link pthread library in your tests? libiomp5.so required pthread library:

g++ test.cpp -o test -L /nfs/site/proj/ipp/build/ipp/ipp70gold-20120406-7-0-0205-105/comp_lib/linux32 -liomp5

/nfs/site/proj/ipp/build/ipp/ipp70gold-20120406-7-0-0205-105/comp_lib/linux32/libiomp5.so: undefined reference to `pthread_atfork'

collect2: ld returned 1 exit status

Pavel

[Francesco_Lamonica]

Hi Pavel,

here is the result with the changed code:

vm-etch-dev32# cat ipp_test_intel_get_procs.cpp

#include

extern "C" {

extern int omp_get_num_procs (void);

}

int main(int argc, char** argv)

{

int p = omp_get_num_procs();

std::cout << "testing ipp: no of procs is" << p << std::endl;

return 0;

}

vm-etch-dev32# g++ -Wall -g ipp_test_intel_get_procs.cpp -L./ -liomp5 -o ipp_test_get_procs

etch:/tmp/ipp-illegal-instruction-test# ./ipp_test_get_procs

Illegal instruction

so it seems that calling omp_get_num_procs triggers the illegal instruction problem (on 7.0.7)

as per the libpthread, i don't know why you get that error, g++ automatically links libpthread on my machines:

etch:/tmp/ipp-illegal-instruction-test# ldd ./ipp_test_get_procs

linux-gate.so.1 => (0xb7fe8000)
libiomp5.so => /tmp/ipp-illegal-instruction-test/libiomp5.so (0xb7f2a000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7e41000)
libm.so.6 => /lib/tls/libm.so.6 (0xb7e1c000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7e11000)
libc.so.6 => /lib/tls/libc.so.6 (0xb7cdf000)
libpthread.so.0 => /lib/tls/libpthread.so.0 (0xb7ccc000)
libdl.so.2 => /lib/tls/libdl.so.2 (0xb7cc8000)
/lib/ld-linux.so.2 (0xb7fe9000)

[Pavel_B_Intel1]

Hi Francesco,

so the problem is in libiomp5 library - it is Intel compiler OpenMP library and we will address the problem to the compiler. Thank you very much for your help!!!

To avoid the problem right now I recommend you to use not threaded variant of IPP libraries: link your application with libipp{domain}_l.a libraries - they don't depend on libiomp5 library, I hope it will be ok for you.

Pavel

[pbkenned1]

Hello Francesco,
It's not clear to me if your Geode processor supports SSE and/or MMX instructions. If it does not, you might try telling the linker(via gcc) that you're compiling for only a Pentium classtarget, by using
-march=pentium.

Do you still see the error if you add '-m32 -march=pentium' to your compilation command?

I don't have a Geode processor system, but on a Pentiumsystem using g++ 4.1, the program runs normally. I also added '-m32' to generate 32-bit code, as the default on my system is to generate 64-bit code. You shouldn't need that on your 32-bit system (but it can't hurt).

> g++ --version g++ (GCC) 4.1.0 (SUSE Linux)

> g++ U105439.cpp -liomp5 -m32 -march=pentium && ./a.out

testing ipp: no of procs is 2

>

Thanks,
Patrick Kennedy
Intel Developer Support

[Francesco_Lamonica]

Hi Patrick,

i will do the test you mentioned, however... if the problems lies in the libiomp5.so library and not in my code, how does that affect the code produced by gcc? does the linker inform somehow the shared lib about the architecture?

thanks

-- Add --

i did the test, nothing changed at all