compiling ifort with -openmp flag

[rdpob]

In Mac OS 10.6, I have a program that compiles and runs fine without the -openmp flag. As soon as I introduce that flag, I can compile fine, but get an immediate crash on the first line of the code with the error "illegal instruction". It doesnt matter whether or not I have any openmp directives in the code, I get the same result.
I've also run a couple of simple examples from the Openmp book and these compile and run fine. Any ideas?

[Steven_L_Intel1]

Does the same thing happen if you use -auto instead of -openmp?

[rdpob]

In my version of ifort, I see no -auto option, other than -auto-scalar. Do you mean -parallel? It runs fine with the -parallel option (though the speed up is perhaps only %20 or so).

[Steven_L_Intel1]

No, I don't mean -parallel. -auto is a valid option. If you find, somehow, that doesn't work, try -recursive.

[rdpob]

I tried both, separately, with the same previous result. While the compiler didnt complain, there is no -auto flag in the man pages for Mac OS X ifort.

Any other ideas? By the way, thanks for taking the time!

-Lee

[rdpob]

Here is the Makefile, if that is of help. Also, in the intel debugger, I get a failure on the first line of the program
"PROGRAM MAIN", with the descriptor "SIGBUS" error.

#
# Makefile for the 3D Shear Velocity inversion
#

FC = ifort
FFLAGS = -parallel -g -openmp -recursive -debug extended -O0 -m64 -openmp-report=1 -align all
EXTND=-extend_source 132

#FC = gfortran
#EXTND=-ffixed-line-length-none

CC = gcc
CFLAGS = -O -m64

#
# You need a sacio lib to link this code.
#

SACLIB = /usr/local/bin/sac/lib/libsacio.a

# Need dynamic libraries, but already set in shell
# DYLD_LIBRARY_PATH = /Developer/opt/intel/composer_xe_2011_sp1.7.246/compiler/lib

#
# Executable will be placed here.
#

BINDIR=.

OBJS = lovdsp.o lovmrx.o raydsp.o raymrx.o partials.o gett.o \
lprModel.o rayrkg.o lovrkg.o drgSubroutines.o \
buildGravitynew.o transform_r_gfortran.o filterGravity.o
#
# Small problems can be handled by an SVD approach
#

SVDOBJS = svdrs.o h12.o

#
# Larger problems require a conjugate gradiewnt-based approach.
#

LSQROBJS = lsqr.o lsqrblas.o aprod.o row_io.o lsqrUtils.o

#
# dr indicates dispersion & rftn inversion
#

dr_3d_inv: $(OBJS) $(SVDOBJS) $(LSQROBJS) model3d.inc dr_3d_inv.o
$(FC) $(FFLAGS) $(OBJS) $(SVDOBJS) $(LSQROBJS) -o $(BINDIR)/dr_3d_inv dr_3d_inv.o $(SACLIB) $(EXTND)

#
clean:
# rm *.o

#
# dependencies
#

dr_3d_inv.o: dr_3d_inv.f model3d.inc
$(FC) $(FFLAGS) $(EXTND) $(OBJS) $(SVDOBJS) $(LSQROBJS) model3d.inc -c dr_3d_inv.f $(SACLIB)
#
lsqrUtils.o: lsqrUtils.f
$(FC) $(FFLAGS) $(EXTND) -c lsqrUtils.f
#
drgSubroutines.o:drgSubroutines.f
$(FC) $(FFLAGS) $(EXTND) -c drgSubroutines.f
#
buildGravitynew.o: buildGravitynew.f model3d.inc
$(FC) $(FFLAGS) $(EXTND) -c buildGravitynew.f
#
filterGravity.o: filterGravity.f model3d.inc
$(FC) $(FFLAGS) $(EXTND) -c filterGravity.f
#
transform_r_gfortran.o:transform_r_gfortran.c
#
svdrs.o: svdrs.f
h12.o: h12.f
lovdsp.o: lovdsp.f
lovmrx.o: lovmrx.f
raydsp.o: raydsp.f
raymrx.o: raymrx.f
partials.o: partials.f
gett.o: gett.f
sac2xy.o: sac2xy.f
srf2sac.o: srf2sac.f
lovrkg.o: lovrkg.f
rayrkg.o: rayrkg.f
lsqr.o: lsqr.f
lsqrblas.o:lsqrblas.f
aprod.o: aprod.c
row_io.o: row_io.c
#

[Steven_L_Intel1]

I wanted to see if -auto caused the same problem as -openmp. Do I understand that it does? If so, then what is likely happening is that you are overflowing the stack. -openmp implies -auto (-recursive is an alias) - both are in the documentation (not sure about man pages, though - I don't think the man page is comprehensive.) This puts all local variables on the stack. OpenMP complicates the issue by having thread-specific stacks.

The first thing I suggest you try is to unlimit the stack. The syntax to do this varies by shell, it can be:

ulimit stack

or

limit stacksize unlimited

This doesn't really make it "unlimited", but raises the size to the max configured for the kernel. You may then need to try setting the environment variable KMP_STACKSIZE to a larger value - try "16M" (without the quotes).

[rdpob]

Is this possibly a stack size issue? I've set OMP_STACKSIZE to sizes up to 1G with no luck. I've also used the csh built-in command ulimit to set it to the hard amount, which I think is 65M (up from 8M). Or are those values given in K?

Any help ideas welcome....

[rdpob]

Steve, do I need to compile with the stack increased or just run in a shell with it increased? Or both?

[TimP]

OMP_STACKSIZE defaults to 4MB (for 64-bit builds). The largest I have seen used is 32MB. This amount, times number of threads, comes out of your total stack, as seen by ulimit.
The build doesn't remember anything about what stack you had set at build time, and there's no stack size built in such as you would set when using Microsoft linker.

[rdpob]

Somehow I didnt get yesterday's reply until today. Oh well. So, I've set OMP_STACKSIZE and KMP_STACKSIZE to 16M and used limit stacksize unlimited in both the run and compile shells. Still fails with same error. I checked and using -automatic gives me "option deprecated". I think it has been replaced with -parallel.

the code runs with -parallel, but fails with both -automatic and -openmp, with or without the stacksize settings.

[rdpob]

It is worth pointing out that the code fails regardless of whether or not I've included any openmp directives in the actual program. It is just the addition of "-openmp" to the compilation line that introduces the errors.

[rdpob]

When I set OMP_NUM_THREADS to 1,2 or 8, (with the stack sizes enlarged as before) I get the same error.

[Steven_L_Intel1]

-auto has nothing to do with -parallel. I didn't say -automatic.

[rdpob]

It doesnt work with -auto either....

[rdpob]

When I look in the Intel Software Development Tools/Intel FOrtran Compiler User and Reference Guides the option "auto" refers one to the option "automatic". Document number 304970-006US. Maybe its an old version, who knows...

[rdpob]

By the way, I should have mentioned this earlier, I'm using XE 12.1.

[Steven_L_Intel1]

Documentation for -auto

If -auto behaves the same way as -openmp, then that confirms my theory that stack space is the issue.

What happens if you add -heap-arrays ?

[rdpob]

Yeah, it still fails in the same way. Thanks for the doc info. I'm slowly getting better at using the documentation...

[rdpob]

Here is a summary of what doesnt work. I've set the stacksize to unlimited (65336) and can check to be sure
that is the case. I've set KMP_STACKSIZE to 16M. I've tried compiling with -auto, -openmp, -recursive, -heap-arrays. So far, none of this has made a difference. I still get an error on the first line when I run the code. In the
debugger (IDB), I get a SIGBUS error. Outside of the debugger, I get "illegal instruction". Anybody have any further ideas? Thanks for your contributions so far!

-Lee

[jimdempseyatthecove]

>>I get a failure on the first line of the program "PROGRAM MAIN", with the descriptor "SIGBUS" error.
...
#
transform_r_gfortran.o:transform_r_gfortran.c
#
...
lsqrblas.o:lsqrblas.f
aprod.o: aprod.c
...
<<

You apparently have a mixed language program. Nothing inherently wrong with that, however if your ".c" programs are C++ they then may have a ctor being executed for a static object which runs prior to the entry to "PROGRAM MAIN". These ctor's (if present) may require some initialization which has not yet been done.

Also, there may (or may not) be an issue of mixing OpenMP in FORTRAN and C/CPP with/without a dependancy of if "main" is in C/CPP or in FORTRAN "PROGRAM ...".

Jim Dempsey