Re: SIGSEGV in JVM when calculating with large matrices

Intel® oneAPI Math Kernel Library

Ask questions and share information with other developers who use Intel® Math Kernel Library.

SIGSEGV in JVM when calculating with large matrices

976 Views

I am having trouble running shared object library that is used by a Java application when the problem size is large. There seems to be a bug somewhere in the interaction between Java and MKL. Hopefully someone has seen this, or can reproduce the bug.

To create the JVM crash, simply modify the source code for the dgemm.java example supplied with the 10.1 MKL to include these lines:

int sz = 4000;
double[] x = new double[sz*sz];
for(int r=0;r = Math.random();
double[] y = new double[sz*sz];
CBLAS.dgemm(Order,TransA,TransB,sz,sz,sz,alpha,x,sz,x,sz,beta,y,sz);
double[] z = new double[sz*sz];
for(int r=0;r = Math.random();
CBLAS.dgemm(Order,TransA,TransB,sz,sz,sz,alpha,z,sz,z,sz,beta,y,sz);

Rebuild the examples, and run with increased memory limits (-Xms1024m -Xmx4096m), and the JVM crashes in the calculation of the second matrix multiplcation. I can generate the same problem with other BLAS routines, and with LAPACK routines too. I have run this with different jdk1.6 releases, and on Red Hat and Ubuntu intel64 systems, and with MKL 10.0, with the same results.

Here is the log from the JVM crash:

#
# An unexpected error has been detected by Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f1f1468869b, pid=25403, tid=1114712400
#
# Java VM: OpenJDK 64-Bit Server VM (1.6.0_0-b11 mixed mode linux-amd64)
# Problematic frame:
# C [libpthread.so.0+0xe69b] raise+0x2b
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-6/
#

--------------- T H R E A D ---------------

Current thread is native thread

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=-6 (FPE_FLTOVF), si_addr=0x000003e80000633b

Registers:
RAX=0x0000000000000000, RBX=0x0000000000000000, RCX=0xffffffffffffffff, RDX=0x000000000000000b
RSP=0x0000000042711c08, RBP=0x0000000042712030, RSI=0x000000000000634d, RDI=0x000000000000633b
R8 =0x00007f1f14c4a384, R9 =0x00007f1f14c4a2f8, R10=0x0000000000000000, R11=0x0000000000000202
R12=0x0000000000000000, R13=0x0000000000000020, R14=0x000000000000000a, R15=0x00007f1f14c4ba80
RIP=0x00007f1f1468869b, EFL=0x0000000000000202, CSGSFS=0x0000000000000033, ERR=0x0000000000000000
TRAPNO=0x0000000000000000

Top of Stack: (sp=0x0000000042711c08)
0x0000000042711c08: 00007f1f14b2c2a8 0000000000000000
0x0000000042711c18: 0000000000000000 0000000000000000
0x0000000042711c28: 0000000000000000 0000000000000000
0x0000000042711c38: 0000000000000000 0000000000000000
0x0000000042711c48: 0000000000000000 0000000000000000
0x0000000042711c58: 0000000000000000 0000000000000000
0x0000000042711c68: 0000000000000000 0000000000000000
0x0000000042711c78: 0000000000000000 0000000000000000
0x0000000042711c88: 0000000000000000 0000000000000000
0x0000000042711c98: 00007f1f14c4ba80 0000000000000000
0x0000000042711ca8: 0000000000000000 0000000000000000
0x0000000042711cb8: 0000000000000000 0000000000000000
0x0000000042711cc8: 0000000000000000 0000000000000000
0x0000000042711cd8: 0000000000000000 0000000000000000
0x0000000042711ce8: 0000000000000000 0000000000000000
0x0000000042711cf8: 0000000000011000 0000000000000000
0x0000000042711d08: 0000000000000000 0000000000000000
0x0000000042711d18: 0000000042702000 0000000000000000
0x0000000042711d28: 0000000000000000 0000000000000000
0x0000000042711d38: 0000000000001000 0000000000000000
0x0000000042711d48: 0000000000000000 0000000000000000
0x0000000042711d58: 0000000000000000 0000000000000008
0x0000000042711d68: 0000000000001000 0000000042713000
0x0000000042711d78: 0000000000011000 000000000073a800
0x0000000042711d88: 0000000000000020 0000000000000000
0x0000000042711d98: 0000000000000000 0000000000000000
0x0000000042711da8: 0000000040abb0f8 0000000000000000
0x0000000042711db8: 0000000000000000 0000000000000000
0x0000000042711dc8: 0000000000000000 0000000000000000
0x0000000042711dd8: 0000000000000000 0000000000000000
0x0000000042711de8: 0000000000000000 0000000000000000
0x0000000042711df8: 0000000000000000 0000000000000000

Instructions: (pc=0x00007f1f1468869b)
0x00007f1f1468868b: f8 1f 31 c7 29 c7 b8 ea 00 00 00 48 63 ff 0f 05
0x00007f1f1468869b: 48 3d 00 f0 ff ff 89 c2 77 03 89 d0 c3 48 8b 05

Stack: [0x0000000042702000,0x0000000042713000], sp=0x0000000042711c08, free space=63k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libpthread.so.0+0xe69b] raise+0x2b

--------------- P R O C E S S ---------------

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000000000060f6b0/0x000000000060f700] Safepoint_lock - owner thread: 0x0000000000613000
[0x000000000060f890/0x000000000060f8e0] Threads_lock - owner thread: 0x0000000000686000

Heap
PSYoungGen total 305856K, used 130500K [0x00007f1eb99b0000, 0x00007f1edef10000, 0x00007f1f0ef00000)
eden space 262208K, 49% used [0x00007f1eb99b0000,0x00007f1ec18e1170,0x00007f1ec99c0000)
from space 43648K, 0% used [0x00007f1ec99c0000,0x00007f1ec9a00030,0x00007f1ecc460000)
to space 43648K, 0% used [0x00007f1edc470000,0x00007f1edc470000,0x00007f1edef10000)
PSOldGen total 699072K, used 125000K [0x00007f1e0ef00000, 0x00007f1e399b0000, 0x00007f1eb99b0000)
object space 699072K, 17% used [0x00007f1e0ef00000,0x00007f1e16912018,0x00007f1e399b0000)
PSPermGen total 21248K, used 2796K [0x00007f1dcef00000, 0x00007f1dd03c0000, 0x00007f1e0ef00000)
object space 21248K, 13% used [0x00007f1dcef00000,0x00007f1dcf1bb090,0x00007f1dd03c0000)

Dynamic libraries:
00400000-00409000 r-xp 00000000 08:01 653763 /usr/lib/jvm/java-6-openjdk/jre/bin/java
00609000-0060a000 rw-p 00009000 08:01 653763 /usr/lib/jvm/java-6-openjdk/jre/bin/java
0060a000-007e3000 rw-p 0060a000 00:00 0 [heap]
40544000-40547000 ---p 40544000 00:00 0
40547000-40645000 rwxp 40547000 00:00 0
40645000-40648000 ---p 40645000 00:00 0
40648000-40746000 rwxp 40648000 00:00 0
40834000-40835000 ---p 40834000 00:00 0
40835000-40935000 rwxp 40835000 00:00 0
409bc000-409bf000 ---p 409bc000 00:00 0
409bf000-40abd000 rwxp 409bf000 00:00 0
419f5000-419f6000 ---p 419f5000 00:00 0
419f6000-41af6000 rwxp 419f6000 00:00 0
41af6000-41af7000 ---p 41af6000 00:00 0
41af7000-41bf7000 rwxp 41af7000 00:00 0
41bf7000-41bf8000 ---p 41bf7000 00:00 0
41bf8000-41cf8000 rwxp 41bf8000 00:00 0
41cf8000-41cf9000 ---p 41cf8000 00:00 0
41cf9000-41df9000 rwxp 41cf9000 00:00 0
41df9000-41dfa000 ---p 41df9000 00:00 0
41dfa000-41efa000 rwxp 41dfa000 00:00 0
41efa000-41efb000 ---p 41efa000 00:00 0
41efb000-41ffb000 rwxp 41efb000 00:00 0
41ffb000-41ffc000 ---p 41ffb000 00:00 0
41ffc000-420fc000 rwxp 41ffc000 00:00 0
420fc000-420fd000 ---p 420fc000 00:00 0
420fd000-421fd000 rwxp 420fd000 00:00 0
421fd000-42200000 ---p 421fd000 00:00 0
42200000-422fe000 rwxp 42200000 00:00 0
422fe000-42301000 ---p 422fe000 00:00 0
42301000-423ff000 rwxp 42301000 00:00 0
423ff000-42402000 ---p 423ff000 00:00 0
42402000-42500000 rwxp 42402000 00:00 0
42500000-42503000 ---p 42500000 00:00 0
42503000-42601000 rwxp 42503000 00:00 0
42601000-42602000 ---p 42601000 00:00 0
42602000-42702000 rwxp 42602000 00:00 0
42702000-42703000 ---p 42702000 00:00 0
42703000-42713000 rwxp 42703000 00:00 0
42713000-42714000 ---p 42713000 00:00 0
42714000-42b14000 rwxp 42714000 00:00 0
42b14000-42b15000 ---p 42b14000 00:00 0
42b15000-42f15000 rwxp 42b15000 00:00 0
42f15000-42f16000 ---p 42f15000 00:00 0
42f16000-43316000 rwxp 42f16000 00:00 0
43316000-43317000 ---p 43316000 00:00 0
43317000-43717000 rwxp 43317000 00:00 0
43717000-43718000 ---p 43717000 00:00 0
43718000-43b18000 rwxp 43718000 00:00 0
43b18000-43b19000 ---p 43b18000 00:00 0
43b19000-43f19000 rwxp 43b19000 00:00 0
43f19000-43f1a000 ---p 43f19000 00:00 0
43f1a000-4431a000 rwxp 43f1a000 00:00 0
7f1da4000000-7f1da4223000 rw-p 7f1da4000000 00:00 0
7f1da4223000-7f1da8000000 ---p 7f1da4223000 00:00 0
7f1da8000000-7f1da8223000 rw-p 7f1da8000000 00:00 0
7f1da8223000-7f1dac000000 ---p 7f1da8223000 00:00 0
7f1dafdfd000-7f1db0227000 rw-p 7f1dafdfd000 00:00 0
7f1db0227000-7f1db4000000 ---p 7f1db0227000 00:00 0
7f1db40dd000-7f1db4aeb000 rw-p 7f1db40dd000 00:00 0
7f1dcb924000-7f1dcc495000 r-xp 00000000 08:01 183882 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so
7f1dcc495000-7f1dcc595000 ---p 00b71000 08:01 183882 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so
7f1dcc595000-7f1dcc67a000 rw-p 00b71000 08:01 183882 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so
7f1dcc67a000-7f1dcc687000 r-xp 00000000 08:01 1210060 /lib/libgcc_s.so.1
7f1dcc687000-7f1dcc887000 ---p 0000d000 08:01 1210060 /lib/libgcc_s.so.1
7f1dcc887000-7f1dcc888000 rw-p 0000d000 08:01 1210060 /lib/libgcc_s.so.1
7f1dcc888000-7f1dcc8c2000 r-xp 00000000 08:01 181292 /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5
7f1dcc8c2000-7f1dcc9c2000 ---p 0003a000 08:01 181292 /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5
7f1dcc9c2000-7f1dcc9c4000 rw-p 0003a000 08:01 181292 /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5
7f1dcc9c4000-7f1dcc9c5000 rw-p 7f1dcc9c4000 00:00 0
7f1dcc9c5000-7f1dcca81000 r-xp 00000000 08:01 181295 /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so
7f1dcca81000-7f1dccb81000 ---p 000bc000 08:01 181295 /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so
7f1dccb81000-7f1dccb82000 rw-p 000bc000 08:01 181295 /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so
7f1dccb82000-7f1dccdd3000 r-xp 00000000 08:01 181293 /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so
7f1dccdd3000-7f1dcced3000 ---p 00251000 08:01 181293 /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so
7f1dcced3000-7f1dcced7000 rw-p 00251000 08:01 181293 /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so
7f1dcced7000-7f1dcced8000 rw-p 7f1dcced7000 00:00 0
7f1dcced8000-7f1dcd71b000 r-xp 00000000 08:01 183907 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_lapack.so
7f1dcd71b000-7f1dcd81a000 ---p 00843000 08:01 183907 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_lapack.so
7f1dcd81a000-7f1dcd824000 rw-p 00842000 08:01 183907 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_lapack.so
7f1dcd824000-7f1dcd851000 r-xp 00000000 08:01 197915 /opt/intel/Compiler/11.0/069/mkl/examples/java/bin/soem64t_intel/libmkl_java_stubs.so
7f1dcd851000-7f1dcda50000 ---p 0002d000 08:01 197915 /opt/intel/Compiler/11.0/069/mkl/examples/java/bin/soem64t_intel/libmkl_java_stubs.so
7f1dcda50000-7f1dcda51000 rw-p 0002c000 08:01 197915 /opt/intel/Compiler/11.0/069/mkl/examples/java/bin/soem64t_intel/libmkl_java_stubs.so
7f1dcda51000-7f1dcda58000 r--s 00000000 08:01 1632007 /usr/lib/gconv/gconv-modules.cache
7f1dcda58000-7f1dcda97000 r--p 00000000 08:01 669150 /usr/lib/locale/en_US.utf8/LC_CTYPE
7f1dcda97000-7f1dcdbf2000 rw-p 7f1dcda97000 00:00 0
7f1dcdbf2000-7f1dcdd81000 r--s 033ba000 08:01 653701 /usr/lib/jvm/java-6-openjdk/jre/lib/rt.jar
7f1dcdd81000-7f1dcdda9000 rw-p 7f1dcdd81000 00:00 0
7f1dcdda9000-7f1dcddb4000 rwxp 7f1dcdda9000 00:00 0
7f1dcddb4000-7f1dcdfa9000 rwxp 7f1dcddb4000 00:00 0
7f1dcdfa9000-7f1dce0ff000 rwxp 7f1dcdfa9000 00:00 0
7f1dce0ff000-7f1dce4ff000 rwxp 7f1dce0ff000 00:00 0
7f1dce4ff000-7f1dce50a000 rwxp 7f1dce4ff000 00:00 0
7f1dce50a000-7f1dce6ff000 rwxp 7f1dce50a000 00:00 0
7f1dce6ff000-7f1dce855000 rwxp 7f1dce6ff000 00:00 0
7f1dce855000-7f1dcec54000 rwxp 7f1dce855000 00:00 0
7f1dcec54000-7f1dced80000 rwxp 7f1dcec54000 00:00 0
7f1dced80000-7f1dceeff000 rwxp 7f1dced80000 00:00 0
7f1dceeff000-7f1dd03c0000 rwxp 7f1dceeff000 00:00 0
7f1dd03c0000-7f1e0ef00000 rwxp 7f1dd03c0000 00:00 0
7f1e0ef00000-7f1e399b0000 rwxp 7f1e0ef00000 00:00 0
7f1e399b0000-7f1eb99b0000 rwxp 7f1e399b0000 00:00 0
7f1eb99b0000-7f1edef10000 rwxp 7f1eb99b0000 00:00 0
7f1edef10000-7f1f0ef00000 rwxp 7f1edef10000 00:00 0
7f1f0ef0e000-7f1f0ef18000 rwxp 7f1f0ef0e000 00:00 0
7f1f0ef18000-7f1f0efce000 rwxp 7f1f0ef18000 00:00 0
7f1f0efce000-7f1f0f23e000 rwxp 7f1f0efce000 00:00 0
7f1f0f23e000-7f1f11fce000 rwxp 7f1f0f23e000 00:00 0
7f1f11fce000-7f1f11fd5000 r-xp 00000000 08:01 653740 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libzip.so
7f1f11fd5000-7f1f121d5000 ---p 00007000 08:01 653740 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libzip.so
7f1f121d5000-7f1f121d6000 rw-p 00007000 08:01 653740 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libzip.so
7f1f121d6000-7f1f12203000 r-xp 00000000 08:01 653719 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libjava.so
7f1f12203000-7f1f12402000 ---p 0002d000 08:01 653719 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libjava.so
7f1f12402000-7f1f12406000 rw-p 0002c000 08:01 653719 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libjava.so
7f1f12406000-7f1f12413000 r-xp 00000000 08:01 653739 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libverify.so
7f1f12413000-7f1f12612000 ---p 0000d000 08:01 653739 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libverify.so
7f1f12612000-7f1f12614000 rw-p 0000c000 08:01 653739 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/libverify.so
7f1f12614000-7f1f1261e000 r-xp 00000000 08:01 1210017 /lib/libnss_files-2.7.so
7f1f1261e000-7f1f1281e000 ---p 0000a000 08:01 1210017 /lib/libnss_files-2.7.so
7f1f1281e000-7f1f12820000 rw-p 0000a000 08:01 1210017 /lib/libnss_files-2.7.so
7f1f12820000-7f1f1282a000 r-xp 00000000 08:01 1210019 /lib/libnss_nis-2.7.so
7f1f1282a000-7f1f12a29000 ---p 0000a000 08:01 1210019 /lib/libnss_nis-2.7.so
7f1f12a29000-7f1f12a2b000 rw-p 00009000 08:01 1210019 /lib/libnss_nis-2.7.so
7f1f12a2b000-7f1f12a33000 r-xp 00000000 08:01 1210015 /lib/libnss_compat-2.7.so
7f1f12a33000-7f1f12c32000 ---p 00008000 08:01 1210015 /lib/libnss_compat-2.7.so
7f1f12c32000-7f1f12c34000 rw-p 00007000 08:01 1210015 /lib/libnss_compat-2.7.so
7f1f12c34000-7f1f12c4a000 r-xp 00000000 08:01 1210014 /lib/libnsl-2.7.so
7f1f12c4a000-7f1f12e49000 ---p 00016000 08:01 1210014 /lib/libnsl-2.7.so
7f1f12e49000-7f1f12e4b000 rw-p 00015000 08:01 1210014 /lib/libnsl-2.7.so
7f1f12e4b000-7f1f12e4d000 rw-p 7f1f12e4b000 00:00 0
7f1f12e61000-7f1f12e69000 r-xp 00000000 08:01 653741 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/native_threads/libhpi.so
7f1f12e69000-7f1f13069000 ---p 00008000 08:01 653741 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/native_threads/libhpi.so
7f1f13069000-7f1f1306a000 rw-p 00008000 08:01 653741 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/native_threads/libhpi.so
7f1f1306a000-7f1f1306b000 rw-p 7f1f1306a000 00:00 0
7f1f1306b000-7f1f13073000 r-xp 00000000 08:01 1210024 /lib/librt-2.7.so
7f1f13073000-7f1f13272000 ---p 00008000 08:01 1210024 /lib/librt-2.7.so
7f1f13272000-7f1f13274000 rw-p 00007000 08:01 1210024 /lib/librt-2.7.so
7f1f13274000-7f1f132f4000 r-xp 00000000 08:01 1210012 /lib/libm-2.7.so
7f1f132f4000-7f1f134f3000 ---p 00080000 08:01 1210012 /lib/libm-2.7.so
7f1f134f3000-7f1f134f5000 rw-p 0007f000 08:01 1210012 /lib/libm-2.7.so
7f1f134f5000-7f1f13c5f000 r-xp 00000000 08:01 653742 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so
7f1f13c5f000-7f1f13e5f000 ---p 0076a000 08:01 653742 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so
7f1f13e5f000-7f1f13ed7000 rw-p 0076a000 08:01 653742 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so
7f1f13ed7000-7f1f13f10000 rw-p 7f1f13ed7000 00:00 0
7f1f13f10000-7f1f14068000 r-xp 00000000 08:01 1210005 /lib/libc-2.7.so
7f1f14068000-7f1f14268000 ---p 00158000 08:01 1210005 /lib/libc-2.7.so
7f1f14268000-7f1f1426b000 r--p 00158000 08:01 1210005 /lib/libc-2.7.so
7f1f1426b000-7f1f1426d000 rw-p 0015b000 08:01 1210005 /lib/libc-2.7.so
7f1f1426d000-7f1f14272000 rw-p 7f1f1426d000 00:00 0
7f1f14272000-7f1f14274000 r-xp 00000000 08:01 1210011 /lib/libdl-2.7.so
7f1f14274000-7f1f14474000 ---p 00002000 08:01 1210011 /lib/libdl-2.7.so
7f1f14474000-7f1f14476000 rw-p 00002000 08:01 1210011 /lib/libdl-2.7.so
7f1f14476000-7f1f1447a000 r-xp 00000000 08:01 653705 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/jli/libjli.so
7f1f1447a000-7f1f14679000 ---p 00004000 08:01 653705 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/jli/libjli.so
7f1f14679000-7f1f1467a000 rw-p 00003000 08:01 653705 /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/jli/libjli.so
7f1f1467a000-7f1f14690000 r-xp 00000000 08:01 1210022 /lib/libpthread-2.7.so
7f1f14690000-7f1f14890000 ---p 00016000 08:01 1210022 /lib/libpthread-2.7.so
7f1f14890000-7f1f14892000 rw-p 00016000 08:01 1210022 /lib/libpthread-2.7.so
7f1f14892000-7f1f14896000 rw-p 7f1f14892000 00:00 0
7f1f14896000-7f1f148ac000 r-xp 00000000 08:01 604120 /usr/lib/libz.so.1.2.3.3
7f1f148ac000-7f1f14aac000 ---p 00016000 08:01 604120 /usr/lib/libz.so.1.2.3.3
7f1f14aac000-7f1f14aad000 rw-p 00016000 08:01 604120 /usr/lib/libz.so.1.2.3.3
7f1f14aad000-7f1f14aae000 rw-p 7f1f14aad000 00:00 0
7f1f14aaf000-7f1f14ab8000 r--s 00065000 08:01 768333 /usr/share/java/gnome-java-bridge.jar
7f1f14ab8000-7f1f14ac0000 rw-s 00000000 08:01 2563301 /tmp/hsperfdata_bruce/25403
7f1f14ac0000-7f1f14ac1000 rwxp 7f1f14ac0000 00:00 0
7f1f14ac1000-7f1f14ac2000 ---p 7f1f14ac1000 00:00 0
7f1f14ac2000-7f1f14b45000 r-xp 00000000 08:01 181334 /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so
7f1f14b45000-7f1f14c44000 ---p 00083000 08:01 181334 /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so
7f1f14c44000-7f1f14c4a000 rw-p 00082000 08:01 181334 /opt/intel/Compiler/11.0/069/lib/intel64/libiomp5.so
7f1f14c4a000-7f1f14c52000 rw-p 7f1f14c4a000 00:00 0
7f1f14c52000-7f1f15747000 r-xp 00000000 08:01 183854 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so
7f1f15747000-7f1f15847000 ---p 00af5000 08:01 183854 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so
7f1f15847000-7f1f15970000 rw-p 00af5000 08:01 183854 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_thread.so
7f1f15970000-7f1f1597b000 rw-p 7f1f15970000 00:00 0
7f1f1597b000-7f1f15bca000 r-xp 00000000 08:01 183909 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so
7f1f15bca000-7f1f15cc9000 ---p 0024f000 08:01 183909 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so
7f1f15cc9000-7f1f15cd0000 rw-p 0024e000 08:01 183909 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_intel_lp64.so
7f1f15cd0000-7f1f15cd6000 rw-p 7f1f15cd0000 00:00 0
7f1f15cd6000-7f1f15cf3000 r-xp 00000000 08:01 1207718 /lib/ld-2.7.so
7f1f15cf9000-7f1f15cfc000 rw-p 7f1f15cf9000 00:00 0
7f1f15cfc000-7f1f15dd1000 r-xp 00000000 08:01 183891 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so
7f1f15dd1000-7f1f15ed0000 ---p 000d5000 08:01 183891 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so
7f1f15ed0000-7f1f15ed6000 rw-p 000d4000 08:01 183891 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so
7f1f15ed6000-7f1f15ef3000 rw-p 7f1f15ed6000 00:00 0
7f1f15ef3000-7f1f15ef5000 rw-p 0001d000 08:01 1207718 /lib/ld-2.7.so
7fff1dedf000-7fff1def4000 rwxp 7ffffffe9000 00:00 0 [stack]
7fff1def4000-7fff1def5000 rw-p 7fffffffe000 00:00 0
7fff1dffe000-7fff1e000000 r-xp 7fff1dffe000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

VM Arguments:
jvm_args: -Xms1024m -Xmx4096m -XX:MaxPermSize=1024m -Djava.library.path=../bin/soem64t_intel/
java_command: dgemm
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/lib/jvm/brjava
CLASSPATH=:/pd/mosek/mosek/5/tools/platform/linux64x86/bin/mosek.jar
PATH=/opt/intel/Compiler/11.0/069/bin/intel64:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/pd/agentController/bin
USERNAME=bruce
LD_LIBRARY_PATH=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/opt/intel/Compiler/11.0/069/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Compiler/11.0/069/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/069/lib/intel64::/pd/agentController/lib:/pd/mosek/mosek/5/tools/platform/linux64x86/bin/
LD_PRELOAD=libmkl_intel_lp64.so:libmkl_intel_thread.so:libmkl_core.so:libiomp5.so
SHELL=/bin/bash
DISPLAY=:0.0

Signal Handlers:
SIGSEGV: [libjvm.so+0x60fac0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x60fac0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x4dac80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x4dac80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x4dac80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x4dac80], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x4de010], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x4dc0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x4dc0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x4dc0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x4dc0a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004

--------------- S Y S T E M ---------------

OS:Ubuntu 8.04 (hardy)
uname:Linux 2.6.24-22-generic #1 SMP Mon Nov 24 19:35:06 UTC 2008 x86_64
libc:glibc 2.7 NPTL 2.7
rlimit: STACK 8192k, CORE 0k, NPROC 212992, NOFILE 1024, AS infinity
load average:1.05 0.37 0.48

CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 7 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 24716736k(22979044k free), swap 42965832k(42965832k free)

vm_info: OpenJDK 64-Bit Server VM (1.6.0_0-b11) for linux-amd64 JRE (1.6.0_0-b11), built on Aug 4 2008 10:09:07 by "buildd" with gcc 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

time: Wed Dec 31 21:26:18 2008
elapsed time: 4 seconds

Link Copied

10 Replies

976 Views

A little more detail. Setting MKL_DISABLE_FAST_MM=1 in my environment, gives a different stack trace:

*** glibc detected *** /pd/java/jdk1.7.0/bin/java: free(): invalid pointer: 0x00007f910c914080 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f926c01408a]
/lib/libc.so.6(cfree+0x8c)[0x7f926c017c1c]
/opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_core.so(mkl_serv_deallocate+0xb2)[0x7f926daaabdc]
======= Memory map: ========
40000000-40001000 r-xp 00000000 08:11 56935883 /pd/java/jdk1.7.0/bin/java
40100000-40101000 rwxp 00000000 08:11 56935883 /pd/java/jdk1.7.0/bin/java
40101000-40122000 rwxp 40101000 00:00 0 [heap]
40122000-40123000 ---p 40122000 00:00 0
40123000-40223000 rwxp 40123000 00:00 0
40223000-40224000 ---p 40223000 00:00 0
40224000-40324000 rwxp 40224000 00:00 0
40324000-40325000 ---p 40324000 00:00 0
40325000-40425000 rwxp 40325000 00:00 0
40425000-40426000 ---p 40425000 00:00 0
40426000-40526000 rwxp 40426000 00:00 0
40526000-40527000 ---p 40526000 00:00 0
40527000-40627000 rwxp 40527000 00:00 0
40627000-40628000 ---p 40627000 00:00 0
40628000-40728000 rwxp 40628000 00:00 0
40728000-40729000 ---p 40728000 00:00 0
40729000-40829000 rwxp 40729000 00:00 0
40829000-4082a000 ---p 40829000 00:00 0
4082a000-4092a000 rwxp 4082a000 00:00 0
4092a000-4092b000 ---p 4092a000 00:00 0
4092b000-40a2b000 rwxp 4092b000 00:00 0
40a2b000-40a2c000 ---p 40a2b000 00:00 0
40a2c000-40a3c000 rwxp 40a2c000 00:00 0
40c46000-40c49000 ---p 40c46000 00:00 0
40c49000-40d47000 rwxp 40c49000 00:00 0
40d47000-40d4a000 ---p 40d47000 00:00 0
40d4a000-40e48000 rwxp 40d4a000 00:00 0
40e48000-40e49000 ---p 40e48000 00:00 0
40e49000-41249000 rwxp 40e49000 00:00 0
41249000-4124a000 ---p 41249000 00:00 0
4124a000-4164a000 rwxp 4124a000 00:00 0
4168f000-41692000 ---p 4168f000 00:00 0
41692000-41790000 rwxp 41692000 00:00 0
41790000-41793000 ---p 41790000 00:00 0
41793000-41891000 rwxp 41793000 00:00 0
41891000-41894000 ---p 41891000 00:00 0
41894000-41992000 rwxp 41894000 00:00 0
419d6000-419d9000 ---p 419d6000 00:00 0
419d9000-41ad7000 rwxp 419d9000 00:00 0
41ad7000-41ada000 ---p 41ad7000 00:00 0
41ada000-41bd8000 rwxp 41ada000 00:00 0
41bd8000-41bd9000 ---p 41bd8000 00:00 0
41bd9000-41cd9000 rwxp 41bd9000 00:00 0
41cd9000-41cda000 ---p 41cd9000 00:00 0
41cda000-420da000 rwxp 41cda000 00:00 0
420da000-420db000 ---p 420da000 00:00 0
420db000-424db000 rwxp 420db000 00:00 0
424db000-424dc000 ---p 424db000 00:00 0
424dc000-428dc000 rwxp 424dc000 00:00 0
428dc000-428dd000 ---p 428dc000 00:00 0
428dd000-42cdd000 rwxp 428dd000 00:00 0
42cdd000-42cde000 ---p 42cdd000 00:00 0
42cde000-430de000 rwxp 42cde000 00:00 0
7f90fc000000-7f90fc024000 rwxp 7f90fc000000 00:00 0
7f90fc024000-7f9100000000 ---p 7f90fc024000 00:00 0
7f9100000000-7f9100223000 rwxp 7f9100000000 00:00 0
7f9100223000-7f9104000000 ---p 7f9100223000 00:00 0
7f9107bfb000-7f9108223000 rwxp 7f9107bfb000 00:00 0
7f9108223000-7f910c000000 ---p 7f9108223000 00:00 0
7f910c30b000-7f910cb17000 rwxp 7f910c30b000 00:00 0
7f910cd1a000-7f9123c53000 rwxp 7f910cd1a000 00:00 0
7f9123c53000-7f91247c4000 r-xp 00000000 08:01 183882 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so
7f91247c4000-7f91248c4000 ---p 00b71000 08:01 183882 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so
7f91248c4000-7f91249a9000 rwxp 00b71000 08:01 183882 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_mc.so
7f91249a9000-7f91249b6000 r-xp 00000000 08:01 1210060 /lib/libgcc_s.so.1
7f91249b6000-7f9124bb6000 ---p 0000d000 08:01 1210060 /lib/libgcc_s.so.1
7f9124bb6000-7f9124bb7000 rwxp 0000d000 08:01 1210060 /lib/libgcc_s.so.1
7f9124bcb000-7f9124c05000 r-xp 00000000 08:01 181292 /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5
7f9124c05000-7f9124d05000 ---p 0003a000 08:01 181292 /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5
7f9124d05000-7f9124d07000 rwxp 0003a000 08:01 181292 /opt/intel/Compiler/11.0/069/lib/intel64/libintlc.so.5
7f9124d07000-7f9124d08000 rwxp 7f9124d07000 00:00 0
7f9124d08000-7f9124dc4000 r-xp 00000000 08:01 181295 /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so
7f9124dc4000-7f9124ec4000 ---p 000bc000 08:01 181295 /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so
7f9124ec4000-7f9124ec5000 rwxp 000bc000 08:01 181295 /opt/intel/Compiler/11.0/069/lib/intel64/libsvml.so
7f9124ec5000-7f9125116000 r-xp 00000000 08:01 181293 /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so
7f9125116000-7f9125216000 ---p 00251000 08:01 181293 /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so
7f9125216000-7f912521a000 rwxp 00251000 08:01 181293 /opt/intel/Compiler/11.0/069/lib/intel64/libimf.so
7f912521a000-7f912521b000 rwxp 7f912521a000 00:00 0
7f912521b000-7f9125a5e000 r-xp 00000000 08:01 183907 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_lapack.so
7f9125a5e000-7f9125b5d000 ---p 00843000 08:01 183907 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_lapack.so
7f9125b5d000-7f9125b67000 rwxp 00842000 08:01 183907 /opt/intel/Compiler/11.0/069/mkl/lib/em64t/libmkl_lapack.so
7f9125b67000-7f9125b94000 r-xp 00000000 08:01 197915 /opt/intel/Compiler/11.0/069/mkl/examples/java/bin/soem64t_intel/libmkl_java_stubs.so
7f9125b94000-7f9125d93000 ---p 0002d000 08:01 197915 /opt/intel/Compiler/11.0/069/mkl/examples/java/bin/soem64t_intel/libmkl_java_stubs.so
7f9125d93000-7f9125d94000 rwxp 0002c000 08:01 197915 /opt/intel/Compiler/11.0/069/mkl/examples/java/bin/soem64t_intel/libmkl_java_stubs.so
7f9125d94000-7f9125e94000 rwxp 7f9125d94000 00:00 0
7f9125e94000-7f9125e9b000 r-xs 00000000 08:01 1632007 /usr/lib/gconv/gconv-modules.cache
7f9125e9b000-7f9125eda000 r-xp 00000000 08:01 669150 /usr/lib/locale/en_US.utf8/LC_CTYPE
7f9125eda000-7f9125fc1000 rwxp 7f9125eda000 00:00 0
7f9125fc1000-7f9126149000 r-xs 02e2c000 08:11 56936099 /pd/java/jdk1.7.0/jre/lib/rt.jar
7f9126149000-7f9126254000 rwxp 7f9126149000 00:00 0
7f9126254000-7f9126449000 rwxp 7f9126254000 00:00 0
7f9126449000-7f912659f000 rwxp 7f9126449000 00:00 0
7f912659f000-7f912699f000 rwxp 7f912659f000 00:00 0
7f912699f000-7f91269aa000 rwxp 7f912699f000 00:00 0
7f91269aa000-7f9126b9f000 rwxp 7f91269aa000 00:00 0
7f9126b9f000-7f9126cf5000 rwxp 7f9126b9f000 00:00 0
7f9126cf5000-7f91270f4000 rwxp 7f9126cf5000 00:00 0
7f91270f4000-7f912719f000 rwxp 7f91270f4000 00:00 0
7f912719f000-7f912739f000 rwxp 7f912719f000 00:00 0
7f912739f000-7f9128860000 rwxp 7f912739f000 00:00 0
7f9128860000-7f91673a0000 rwxp 7f9128860000 00:00 0
7f91673a0000-7f9191e50000 rwxp 7f91673a0000 00:00 0
7f9191e50000-7f9211e50000 rwxp 7f9191e50000 00:00 0
7f9211e50000-7f92273a0000 rwxp 7f9211e50000 00:00 0
7f92273a0000-7f92673a0000 rwxp 7f92273a0000 00:00 0
7f92673a9000-7f92673b3000 rwxp 7f92673a9000 00:00 0
7f92673b3000-7f9267469000 rwxp 7f92673b3000 00:00 0
7f9267469000-7f92676d9000 rwxp 7f9267469000 00:00 0
7f92676d9000-7f926a469000 rwxp 7f92676d9000 00:00 0
7f926a469000-7f926a477000 r-xp 00000000 08:11 56951154 /pd/java/jdk1.7.0/jre/lib/amd64/libzip.so
7f926a477000-7f926a579000 ---p 0000e000 08:11 56951154 /pd/java/jdk1.7.0/jre/lib/amd64/libzip.so
7f926a579000-7f926a57d000 rwxp 00010000 08:11 56951154 /pd/java/jdk1.7.0/jre/lib/amd64/libzip.so
7f926a57d000-7f926a587000 r-xp 00000000 08:01 1210017 /lib/libnss_files-2.7.so
7f926a587000-7f926a787000 ---p 0000a000 08:01 1210017 /lib/libnss_files-2.7.so
7f926a787000-7f926a789000 rwxp 0000a000 08:01 1210017 /lib/libnss_files-2.7.so
7f926a789000-7f926a793000 r-xp 00000000 08:01 1210019 /lib/libnss_nis-2.7.so
7f926a793000-7f926a992000 ---p 0000a000 08:01 1210019 /lib/libnss_nis-2.7.so
7f926a992000-7f926a994000 rwxp 00009000 08:01 1210019 /lib/libnss_nis-2.7.so
7f926a994000-7f926a99c000 r-xp 00000000 08:01 1210015 /lib/libnss_compat-2.7.so
7f926a99c000-7f926ab9b000 ---p 00008000 08:01 1210015 /lib/libnss_compat-2.7.soAborted

The code will complete successfully about 2/3 of the time using the prerelease jdk1.7.

Copy link

976 Views

I had exactly the same error show up in another vendor's software, the only overlap with MKL is that they use Intel's libguide library. The vendor was able to reproduce the error. It seems libguide does some delayed thread cleanup and if Java is accessing memory when that happens, it crashes the JVM. I'm not sure if this a bug or "feature" of libguide, or if it could be considered a JVM bug, either way it seems threaded MKL cannot safely be used on Linux from Java. Very disappoitning

Copy link

976 Views

Hello,
Thank you for your comment.
Can you clarify on few items:
1. What do you mean another vendor software ?

2. In fact we tried similar test case with size up to 8000 and have not seen any problems. We suspect that such issues could happen when there is not enough memory for Java.
Can you may be try increase amount of memory to JVM smth as Xms1024m -Xmx4096m
Options depends on JVM you use

Below the environment we used for some experiments with this case:
+++++++++++++++++++
java -version
java version "1.6.0"
Java SE Runtime Environment (build 1.6.0-b105)
Java HotSpot 64-Bit Server VM (build 1.6.0-b105, mixed mode)

icc -V
Intel C Compiler for applications running on Intel 64, Version 10.1 Build 20080801 Package ID: l_cc_p_10.1.018
Copyright (C) 1985-2008 Intel Corporation. All rights reserved.

SLES 9.2 kernel 2.6.5-7.191-smp glibc-2.3.3-98.47
uname -a
Linux nstmklel25 2.6.5-7.191-smp #1 SMP Tue Jun 28 14:58:56 UTC 2005 x86_64 x86_64 x86_64 GNU/Linux

Ubuntu 7.10 kernel 2.6.22-14-server
uname -a
Linux nstmklel32 2.6.22-14-server #1 SMP Sun Oct 14 22:09:15 GMT 2007 x86_64 GNU/Linux
+++++++++++++++++++
3. What OS do you use?
4 have you ever saw the same crashes when used the sequential version of MKL library was used?

--Gennady

Copy link

976 Views

Quoting - Gennady Fedorov (Intel)

Hi , thanks for the repsonse. Here are the answers to your questions:
1. The other software that I used that had exactly the same crash, and which was reproduced by the vendor, was the MOSEK optimizer. For that, and for the MKL, if no threading is used, the crash does not occur.

2. I have run with -Xms1g -Xmx12g on the test problem I described, and it made no difference. You can see in the crash log Iposted that there is free memory at the time of the crash. Memory exhaustion though is a catchable exception, should not produce a JVM crash. I can change the sz parameter in my example to a larger value, and have the JVMnot crash as well. Since it seems to be a memory corruption problem, it is a bit of a crapshoot for when it will actually fail. If I, for example, set ReservedCodeCacheSize=2g that example never fails, others still do though.

3. I have had the same problem on both Ubuntu 8.04, with JDK 1.60_05, 1.6.0_07, and 1.6.0_10 64 bit server on a Xeon 5420, and on Red Hat 5 (glibc-2.5) using JDK 1.6.0_10 and JDK 1.6.0_11. My red Hat machine is Xeon 5410 with kernel version 2.6.18-92.1.22.e15, and my java version is Java HotSpot 64 bit Server (build 11.0-b16, mixed mode). )I see your test system has a pretty dated kernel and glibc version. Since there is always a crash in libpthread, could you try and run on a machine with glibc-2.5 or later?

4. If I use mkl_sequential and set MKL_NUM_THREADS=1 it does not crash, ever.

Copy link

976 Views

I see you guys don't think I'm completely crazy as there is a new "export OMP_NUM_THREADS=1" in the examples makefile of the 10.1.1 release that went out last night. Examples still crashes for me though with this new release.

Copy link

976 Views

This is only a stopgap solution of another (unstable) problem.

-Vladimir

Copy link

976 Views

After having similar issues with ATLAS and MOSEK, I switched to IBM's JVM and have had no problems with that JVM. Everything runs great, no crashes with MKL threaded or unthreaded.

Copy link

976 Views

Hello,
Thank you for the comment. So this is Open JDK issue.
-Vladimir

Copy link

976 Views

Hi,

maybe I have a simple solution for you that works:

http://root.cern.ch/phpBB2/viewtopic.php?p=34767&sid=b0f3674babd977efc75ab70f57489816

I had exactly the same problem, and it has to do with signals from intel mkl that get misinterpreted by jvm,
which then crashes suddenly.

The solution: define LD_PRELOAD=/libjsig.so before you call the jvm.

I hope this helps you - I spent several weeks with solving the problem!

Best regards,
Andreas.

Copy link

976 Views

Hi, Andreas,

Thank you for the recommendation. It's really hard to find the solution and such an opportunity can be used as the workaround.

-Vladimir

Copy link

Community support is provided during standard business hours (Monday to Friday 7AM - 5PM PST). Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.