nodeperf makes SegFault

[Ilya_Zelenchuk]

Good day Everyone!

I've download and install Intel C Compilers & MKL 11.064. Then build OpenMPI with intel compilers. I wish to run nodeperf on my new cluster, so, I compile nodeperf.c from MKL benchmark as:

mpicc -o nodeperf nodeperf.c ../../lib/em64t/libmkl_solver_ilp64_sequential.a -Wl,--start-group ../../lib/em64t/libmkl_intel_ilp64.a ../../lib/em64t/libmkl_sequential.a ../../lib/em64t/libmkl_core.a ../../lib/em64t/libmkl_blacs_openmpi_ilp64.a -Wl,--end-group -lpthread

But when I try to run on single node it crash with a segfault:

No multi-threaded MPI detected (Hybrid may or may not work)

The time/date of the run... at Mon Feb 1 13:11:57 2010

This driver was compiled with:

-DITER=4 -DLINUX -DNOACCUR -DPREC=double

Malloc done. Used 827090096 bytes

[umt:22585] *** Process received signal ***

[umt:22585] Signal: Segmentation fault (11)

[umt:22585] Signal code: (128)

[umt:22585] Failing at address: (nil)

[umt:22585] [ 0] /lib64/libpthread.so.0 [0x3c3600e930]

[umt:22585] [ 1] ./nodeperf(mkl_blas_mc3_dgemm_copybn+0x1c1) [0x54d641]

[umt:22585] [ 2] ./nodeperf(mkl_blas_mc3_xdgemm_par+0xf24) [0x547344]

[umt:22585] [ 3] ./nodeperf(mkl_blas_mc3_xdgemm+0x1c8) [0x4211d8]

[umt:22585] [ 4] ./nodeperf(mkl_blas_xdgemm+0x1c5) [0x406e55]

[umt:22585] [ 5] ./nodeperf(mkl_blas_dgemm+0x51) [0x406b51]

[umt:22585] [ 6] ./nodeperf(DGEMM+0x13a) [0x40685a]

[umt:22585] [ 7] ./nodeperf(main+0x530) [0x4060e0]

[umt:22585] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3c3541d994]

[umt:22585] [ 9] ./nodeperf(realloc+0xb1) [0x405af9]

[umt:22585] *** End of error message ***

Segmentation fault

The node config:

Processors: Intel Xeon CPU E5520 @ 2.27GHz

RAM: 16Gb

MKLROOT and other PATH's looks fine.

Can you help me ?

[Gennady_F_Intel]

Ilya,

Please try to use/DMKL_ILP64 compiler option while linking ilp64 libraries? will it help you?

--Gennady

[Ilya_Zelenchuk]

Seems not. I hope thatoption DMKL_ILP64 is using correct by me.

[root@umt mp_linpack]# mpicc -o nodeperf nodeperf.c -DMKL_ILP64 ../../lib/em64t/libmkl_solver_ilp64_sequential.a -Wl,--start-group ../../lib/em64t/libmkl_intel_ilp64.a ../../lib/em64t/libmkl_sequential.a ../../lib/em64t/libmkl_core.a ../../lib/em64t/libmkl_blacs_openmpi_ilp64.a -Wl,--end-group -lpthread

/opt/intel/Compiler/11.1/064/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail

[root@umt mp_linpack]# ./nodeperf

--------------------------------------------------------------------------

A deprecated MCA parameter value was specified in an MCA parameter

file. Deprecated MCA parameters should be avoided; they may disappear

in future releases.

Deprecated parameter: pls_rsh_agent

--------------------------------------------------------------------------

No multi-threaded MPI detected (Hybrid may or may not work)

The time/date of the run... at Mon Feb 1 17:26:07 2010

This driver was compiled with:

-DITER=4 -DLINUX -DNOACCUR -DPREC=double

Malloc done. Used 827090096 bytes

[umt:24196] *** Process received signal ***

[umt:24196] Signal: Segmentation fault (11)

[umt:24196] Signal code: (128)

[umt:24196] Failing at address: (nil)

[umt:24196] [ 0] /lib64/libpthread.so.0 [0x3c3600e930]

[umt:24196] [ 1] ./nodeperf(mkl_blas_mc3_dgemm_copybn+0x1c1) [0x54d641]

[umt:24196] [ 2] ./nodeperf(mkl_blas_mc3_xdgemm_par+0xf24) [0x547344]

[umt:24196] [ 3] ./nodeperf(mkl_blas_mc3_xdgemm+0x1c8) [0x4211d8]

[umt:24196] [ 4] ./nodeperf(mkl_blas_xdgemm+0x1c5) [0x406e55]

[umt:24196] [ 5] ./nodeperf(mkl_blas_dgemm+0x51) [0x406b51]

[umt:24196] [ 6] ./nodeperf(DGEMM+0x13a) [0x40685a]

[umt:24196] [ 7] ./nodeperf(main+0x530) [0x4060e0]

[umt:24196] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3c3541d994]

[umt:24196] [ 9] ./nodeperf(realloc+0xb1) [0x405af9]

[umt:24196] *** End of error message ***

Segmentation fault

[Alex_Kosenkov]

Hello, Ilya

The answer to this problem is quite simple: MP LINPACK doesn't support ILP64.

You can try to compile it like this:

mpiicc -o nodeperf nodeperf.c -O3 -Wl,--start-group $MKL/libmkl_intel_lp64.a $MKL/libmkl_sequential.a $MKL/libmkl_core.a -Wl,--end-group -lpthread

This way it doesn't crash.

Also don't forget that if you are using mpicc - most likely you are using MPI wrapper for gcc. I don't see the point why not to use mpiicc which is a wrapper for icc.

Hope I've been of any help.

Best regards,

Alexander

[Ilya_Zelenchuk]

Yes, it works now!

Thanks.

I know mpicc is wrapper. But i'm using icc in it.

[root@umt mp_linpack]# mpicc --version

icc (ICC) 11.1 20091130

Copyright (C) 1985-2009 Intel Corporation. All rights reserved.

[root@umt mp_linpack]# mpicc -O3 -o nodeperf nodeperf.c -Wl,--start-group ../../lib/em64t/libmkl_intel_lp64.a ../../lib/em64t/libmkl_sequential.a ../../lib/em64t/libmkl_core.a -Wl,--end-group -lpthread

/opt/intel/Compiler/11.1/064/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail

[root@umt mp_linpack]# taskset 08 ./nodeperf

--------------------------------------------------------------------------

A deprecated MCA parameter value was specified in an MCA parameter

file. Deprecated MCA parameters should be avoided; they may disappear

in future releases.

Deprecated parameter: pls_rsh_agent

--------------------------------------------------------------------------

No multi-threaded MPI detected (Hybrid may or may not work)

The time/date of the run... at Mon Feb 1 19:09:27 2010

This driver was compiled with:

-DITER=4 -DLINUX -DNOACCUR -DPREC=double

Malloc done. Used 827090096 bytes

(0 of 1): NN lda=10000 ldb= 168 ldc=10000 0 0 0 9342.140 x.x.x.x

[root@umt mp_linpack]#

[Alex_Kosenkov]

I'm glad I was helpful.

Best regards,

Alexander