topic compiling ifort with -openmp flag in Intel® Fortran Compiler

compiling ifort with -openmp flag

rdpob — Tue, 21 Feb 2012 16:46:43 GMT

In Mac OS 10.6, I have a program that compiles and runs fine without the -openmp flag. As soon as I introduce that flag, I can compile fine, but get an immediate crash on the first line of the code with the error "illegal instruction". It doesnt matter whether or not I have any openmp directives in the code, I get the same result.
I've also run a couple of simple examples from the Openmp book and these compile and run fine. Any ideas?

compiling ifort with -openmp flag

Steven_L_Intel1 — Tue, 21 Feb 2012 18:30:47 GMT

Does the same thing happen if you use -auto instead of -openmp?

compiling ifort with -openmp flag

rdpob — Tue, 21 Feb 2012 20:22:37 GMT

In my version of ifort, I see no -auto option, other than -auto-scalar. Do you mean -parallel? It runs fine with the -parallel option (though the speed up is perhaps only %20 or so).

compiling ifort with -openmp flag

Steven_L_Intel1 — Tue, 21 Feb 2012 22:11:04 GMT

No, I don't mean -parallel. -auto is a valid option. If you find, somehow, that doesn't work, try -recursive.

compiling ifort with -openmp flag

rdpob — Tue, 21 Feb 2012 22:37:58 GMT

I tried both, separately, with the same previous result. While the compiler didnt complain, there is no -auto flag in the man pages for Mac OS X ifort.

Any other ideas? By the way, thanks for taking the time!

-Lee

compiling ifort with -openmp flag

rdpob — Wed, 22 Feb 2012 16:20:53 GMT

Here is the Makefile, if that is of help. Also, in the intel debugger, I get a failure on the first line of the program
"PROGRAM MAIN", with the descriptor "SIGBUS" error.

#
# Makefile for the 3D Shear Velocity inversion
#

FC = ifort
FFLAGS = -parallel -g -openmp -recursive -debug extended -O0 -m64 -openmp-report=1 -align all
EXTND=-extend_source 132

#FC = gfortran
#EXTND=-ffixed-line-length-none

CC = gcc
CFLAGS = -O -m64

#
# You need a sacio lib to link this code.
#

SACLIB = /usr/local/bin/sac/lib/libsacio.a

# Need dynamic libraries, but already set in shell
# DYLD_LIBRARY_PATH = /Developer/opt/intel/composer_xe_2011_sp1.7.246/compiler/lib

#
# Executable will be placed here.
#

BINDIR=.

OBJS = lovdsp.o lovmrx.o raydsp.o raymrx.o partials.o gett.o \
lprModel.o rayrkg.o lovrkg.o drgSubroutines.o \
buildGravitynew.o transform_r_gfortran.o filterGravity.o
#
# Small problems can be handled by an SVD approach
#

SVDOBJS = svdrs.o h12.o

#
# Larger problems require a conjugate gradiewnt-based approach.
#

LSQROBJS = lsqr.o lsqrblas.o aprod.o row_io.o lsqrUtils.o

#
# dr indicates dispersion & rftn inversion
#

dr_3d_inv: $(OBJS) $(SVDOBJS) $(LSQROBJS) model3d.inc dr_3d_inv.o
$(FC) $(FFLAGS) $(OBJS) $(SVDOBJS) $(LSQROBJS) -o $(BINDIR)/dr_3d_inv dr_3d_inv.o $(SACLIB) $(EXTND)

#
clean:
# rm *.o

#
# dependencies
#

dr_3d_inv.o: dr_3d_inv.f model3d.inc
$(FC) $(FFLAGS) $(EXTND) $(OBJS) $(SVDOBJS) $(LSQROBJS) model3d.inc -c dr_3d_inv.f $(SACLIB)
#
lsqrUtils.o: lsqrUtils.f
$(FC) $(FFLAGS) $(EXTND) -c lsqrUtils.f
#
drgSubroutines.o:drgSubroutines.f
$(FC) $(FFLAGS) $(EXTND) -c drgSubroutines.f
#
buildGravitynew.o: buildGravitynew.f model3d.inc
$(FC) $(FFLAGS) $(EXTND) -c buildGravitynew.f
#
filterGravity.o: filterGravity.f model3d.inc
$(FC) $(FFLAGS) $(EXTND) -c filterGravity.f
#
transform_r_gfortran.o:transform_r_gfortran.c
#
svdrs.o: svdrs.f
h12.o: h12.f
lovdsp.o: lovdsp.f
lovmrx.o: lovmrx.f
raydsp.o: raydsp.f
raymrx.o: raymrx.f
partials.o: partials.f
gett.o: gett.f
sac2xy.o: sac2xy.f
srf2sac.o: srf2sac.f
lovrkg.o: lovrkg.f
rayrkg.o: rayrkg.f
lsqr.o: lsqr.f
lsqrblas.o:lsqrblas.f
aprod.o: aprod.c
row_io.o: row_io.c
#

compiling ifort with -openmp flag

Steven_L_Intel1 — Wed, 22 Feb 2012 18:37:10 GMT

I wanted to see if -auto caused the same problem as -openmp. Do I understand that it does? If so, then what is likely happening is that you are overflowing the stack. -openmp implies -auto (-recursive is an alias) - both are in the documentation (not sure about man pages, though - I don't think the man page is comprehensive.) This puts all local variables on the stack. OpenMP complicates the issue by having thread-specific stacks.

The first thing I suggest you try is to unlimit the stack. The syntax to do this varies by shell, it can be:

ulimit stack

or

limit stacksize unlimited

This doesn't really make it "unlimited", but raises the size to the max configured for the kernel. You may then need to try setting the environment variable KMP_STACKSIZE to a larger value - try "16M" (without the quotes).

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 17:13:15 GMT

Is this possibly a stack size issue? I've set OMP_STACKSIZE to sizes up to 1G with no luck. I've also used the csh built-in command ulimit to set it to the hard amount, which I think is 65M (up from 8M). Or are those values given in K?

Any help ideas welcome....

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 17:16:29 GMT

Steve, do I need to compile with the stack increased or just run in a shell with it increased? Or both?

compiling ifort with -openmp flag

TimP — Thu, 23 Feb 2012 17:32:04 GMT

OMP_STACKSIZE defaults to 4MB (for 64-bit builds). The largest I have seen used is 32MB. This amount, times number of threads, comes out of your total stack, as seen by ulimit.
The build doesn't remember anything about what stack you had set at build time, and there's no stack size built in such as you would set when using Microsoft linker.

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 17:32:29 GMT

Somehow I didnt get yesterday's reply until today. Oh well. So, I've set OMP_STACKSIZE and KMP_STACKSIZE to 16M and used limit stacksize unlimited in both the run and compile shells. Still fails with same error. I checked and using -automatic gives me "option deprecated". I think it has been replaced with -parallel.

the code runs with -parallel, but fails with both -automatic and -openmp, with or without the stacksize settings.

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 17:35:26 GMT

It is worth pointing out that the code fails regardless of whether or not I've included any openmp directives in the actual program. It is just the addition of "-openmp" to the compilation line that introduces the errors.

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 17:49:39 GMT

When I set OMP_NUM_THREADS to 1,2 or 8, (with the stack sizes enlarged as before) I get the same error.

compiling ifort with -openmp flag

Steven_L_Intel1 — Thu, 23 Feb 2012 18:28:37 GMT

-auto has nothing to do with -parallel. I didn't say -automatic.

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 20:39:12 GMT

It doesnt work with -auto either....

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 20:43:16 GMT

When I look in the Intel Software Development Tools/Intel FOrtran Compiler User and Reference Guides the option "auto" refers one to the option "automatic". Document number 304970-006US. Maybe its an old version, who knows...

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 20:45:05 GMT

By the way, I should have mentioned this earlier, I'm using XE 12.1.

compiling ifort with -openmp flag

Steven_L_Intel1 — Thu, 23 Feb 2012 22:02:32 GMT

Documentation for -auto

If -auto behaves the same way as -openmp, then that confirms my theory that stack space is the issue.

What happens if you add -heap-arrays ?

compiling ifort with -openmp flag

rdpob — Thu, 23 Feb 2012 22:19:19 GMT

Yeah, it still fails in the same way. Thanks for the doc info. I'm slowly getting better at using the documentation...

compiling ifort with -openmp flag

rdpob — Tue, 28 Feb 2012 21:27:07 GMT

Here is a summary of what doesnt work. I've set the stacksize to unlimited (65336) and can check to be sure
that is the case. I've set KMP_STACKSIZE to 16M. I've tried compiling with -auto, -openmp, -recursive, -heap-arrays. So far, none of this has made a difference. I still get an error on the first line when I run the code. In the
debugger (IDB), I get a SIGBUS error. Outside of the debugger, I get "illegal instruction". Anybody have any further ideas? Thanks for your contributions so far!

-Lee