Fortran code + Open MP + Open MPI not working

[Sangamesh_B_]

Hi,

The compilation of a fortran application - CPMD-3.13.2 - with OpenMP + OpenMPI-1.3.3 + ifort-10.1 + MKL-10.0 is failing with following error on a Rocks-5.1 Linux cluster:

/lib/cpp -P -C -traditional -D__Linux -D__PGI -DFFT_DEFAULT -DPOINTER8
-DLINUX_IFC -DPARALLEL -DMYRINET ./potfor.F ./potfor.f
/opt/mpi/openmpi/1.3.3/intel/bin/mpif90 -c -pc64 -O3 -xT -unroll -openmp
./potfor.f -o ./potfor.o
fortcom: Severe: **Internal compiler error: segmentation violation signal
raised** Please report this error along with the circumstances in which it
occurred in a Software Problem Report. Note: File and line given may not
be explicit cause of this error.

compilation aborted for ./potfor.f (code 3)
make: *** [potfor.o] Error 3


But if the potfor.f is compiled by removing -openmp option, then it compiles
successfully. But while running it again failed with "Aborted"
message.

The same error is discussed in the following link:

http://www.cpmd.org/pipermail/cpmd-list/2009-April.txt

Is this a CPMD coding problem or ifort or OpenMPI problem?

Please help me to resolve this

Thank you

[Gergana_S_Intel]

Hi sangamesh,

I'll go ahead and move this over the Intel Fortran Compilers forum. Since you say this works without the -openmp flag, they might be able to help there. I would also suggest checking out theHelp/Support section of the Open MPI webpage as they would be the most qualified to answer Open MPI questions.

Also, just as a quick note: the link you provide requires a login. Could you summarize or copy/paste the content from the CPMD thread?

Regards,
~Gergana

[Sangamesh_B_]

Quoting - Gergana Slavova (Intel)

Hi sangamesh,

I'll go ahead and move this over the Intel Fortran Compilers forum. Since you say this works without the -openmp flag, they might be able to help there. I would also suggest checking out theHelp/Support section of the Open MPI webpage as they would be the most qualified to answer Open MPI questions.

Also, just as a quick note: the link you provide requires a login. Could you summarize or copy/paste the content from the CPMD thread?

Regards,
~Gergana

Hi,

I've posted this to Open MPI mailing list:

http://www.open-mpi.org/community/lists/users/2009/10/10985.php

I got to know that, the same code works well with Intel MPI-3.1 on another cluster with same environment but only with slight change in ifort version.
ifort:
in our cluster
/opt/intel/fce/10.1.008/bin/ifort

in their cluster
/opt/intel/fce/10.1.015/bin/ifort

Also I tried with Intel MPI-3.2.2 which also failed with same error:

http://www.cpmd.org/pipermail/cpmd-list/2009-April.txt

If this link is not opening, then copy paste the URL in a web-browser.

Thanks,
Sangamesh

[TimP]

If you compared notes with the openmpi people and find that ifort 10.1.008 had a bug which was fixed in 10.1.015, it should not be difficult to conclude that your compiler is too old.

[Sangamesh_B_]

Quoting - tim18

If you compared notes with the openmpi people and find that ifort 10.1.008 had a bug which was fixed in 10.1.015, it should not be difficult to conclude that your compiler is too old.

With the new version of ifort-11.1.059 the cpmd code is getting compiled but during execution its failing with following error:

$ /opt/mpi/openmpi/1.3.3/intel-11.1.059/bin/mpirun -np 2 /opt/apps/cpmd/3.13.2/openmpi-mkl/cpmd.x 3-proton-md.inp > 3-proton-md.out
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread.so.0 00000037C960DE80 Unknown Unknown Unknown
libpthread.so.0 00000037C960DD5D Unknown Unknown Unknown
cpmd.x 0000000000AA95EF Unknown Unknown Unknown
libpthread.so.0 00000037C9606307 Unknown Unknown Unknown
libc.so.6 00000037C8AD1DED Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread.so.0 00000037C960DE80 Unknown Unknown Unknown
libmkl_mc.so 00002AAAB6FFABDA Unknown Unknown Unknown
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 16670 on
node geetanjali.intranet.jncasr.ac.in exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Did you suggest this version/release of ifort or next release?

Thank you

[TimP]

This looks a lot like a stack overflow. Of course, you have to investigate to confirm whether that is the case. If setting a high stack size limit in your shell, or compiling with -heap-arrays helps, you have confirmed the typical solutions for stack overflow.

[Ron_Green]

Quoting - sangamesh

With the new version of ifort-11.1.059 the cpmd code is getting compiled but during execution its failing with following error:

$ /opt/mpi/openmpi/1.3.3/intel-11.1.059/bin/mpirun -np 2 /opt/apps/cpmd/3.13.2/openmpi-mkl/cpmd.x 3-proton-md.inp > 3-proton-md.out
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread.so.0 00000037C960DE80 Unknown Unknown Unknown
libpthread.so.0 00000037C960DD5D Unknown Unknown Unknown
cpmd.x 0000000000AA95EF Unknown Unknown Unknown
libpthread.so.0 00000037C9606307 Unknown Unknown Unknown
libc.so.6 00000037C8AD1DED Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread.so.0 00000037C960DE80 Unknown Unknown Unknown
libmkl_mc.so 00002AAAB6FFABDA Unknown Unknown Unknown
--------------------------------------------------------------------------
mpirun has exited due to process rank 0 with PID 16670 on
node geetanjali.intranet.jncasr.ac.in exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Did you suggest this version/release of ifort or next release?

Thank you

http://software.intel.com/en-us/articles/determining-root-cause-of-sigsegv-or-sigbus-errors/

[Sangamesh_B_]

Quoting - Ronald W. Green (Intel)

http://software.intel.com/en-us/articles/determining-root-cause-of-sigsegv-or-sigbus-errors/

I didn't use "-heap-arrays" option as OpenMP is used.

Still it segfaults eventhough stack size is set:

$ /opt/apps/cpmd/3.13.2/openmpi-mkl/cpmd.x 3-proton-md.inp > 3-proton-md.out
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread.so.0 00000037C960DE80 Unknown Unknown Unknown
libpthread.so.0 00000037C960DD5D Unknown Unknown Unknown
libiomp5.so 00002AAAACC35EFA Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread.so.0 00000037C960DE80 Unknown Unknown Unknown
libmkl_mc.so 00002AAAB2E10BDA Unknown Unknown Unknown

$ ulimit -s 99999999

$ /opt/apps/cpmd/3.13.2/openmpi-mkl/cpmd.x 3-proton-md.inp > 3-proton-md.out
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpthread.so.0 00000037C960DE80 Unknown Unknown Unknown
libmkl_mc.so 00002AAAB24F8BDA Unknown Unknown Unknown

The compilation flags used are:

FFLAGS = -c -openmp -w90 -w95 -O2 -unroll -ip -march=pentium3 -cm

MKLPATH=/opt/intel/Compiler/11.1/059/mkl/lib/em64t

LFLAGS=-L$(MKLPATH) $(MKLPATH)/libmkl_solver_ilp64.a -Wl,--start-group -lmkl_intel_ilp64 -lmkl_intel_thread -lmkl_core -Wl,--end-group -openmp -lpthread

CFLAGS = -c -openmp -O2 -Wall

Is this segfault due to mkl? or ifort?

Thanks,
Sangamesh

[TimP]

Quoting - sangamesh

Is this segfault due to mkl? or ifort?

How about asking yourself this question? If setting -traceback in the link doesn't give you automatic full stack trace, running under gdb or idb should do so. You have the controls to enable OpenMP independently in MKL and in your program; we don't.
If you are exceeding the default thread stack size (probably 2MB for 32-bit, 4MB for 64-bit compilation), the KMP_STACKSIZE environment variables and function calls are available to adjust it, along with setting your overall stack size.

[Sangamesh_B_]

From the CPMD mailing list, I came to know that the code is able to compile with ifort-11.0 version. So can you please provide me the download link of ifort-11.0 version?

http://www.cpmd.org/pipermail/cpmd-list/2009-November/004918.html

Thanks

[Ron_Green]

Quoting - sangamesh

From the CPMD mailing list, I came to know that the code is able to compile with ifort-11.0 version. So can you please provide me the download link of ifort-11.0 version?

http://www.cpmd.org/pipermail/cpmd-list/2009-November/004918.html

Thanks

http://software.intel.com/en-us/articles/older-version-product/