What -openmp does without any !$OMP directives

[juansheng]

Hi there,

I wrote a simple fortran 90 project with some modules involved. What supprising me is that building with -openmp option, program generates totally wrong results while there weren't any !$OMP directives in this project. Could anybody tell me what the option '-openmp' of ifort does when any codes haven't been embraced with !$OMP directives? Thanks.

Regards,

[mecej4]

Please show an example where this happens. It is possible that your project uses (explicitly or implicitly) libraries that use OpenMP, and the -openmp option affects the behavior of routines linked in from those libraries and, therefore, of your application.

[TimP]

Adding -openmp without directives includes some initialization code. Changes in memory layout may have surfaced problems with undefined data.

[pbkenned1]

What happens if you compile WITHOUT -openmp, but WITH -auto?

As Tim said, -openmp changes the memory model. Compiling with -openmp implies -auto, meaning all local, non-SAVEd variables (scalars and arrays of all types) will be allocated on the runtime stack.

With -auto, you may get different numerical results because you may have been implicitly relying on static storage to preserve values between procedure calls. Compiling with just -auto essentially gives you the OpenMP memory model.

Patrick Kennedy

[Martyn_C_Intel]

I suggest that you compile with -check uninit-traceback (along with either -auto or -openmp) and run your application. You should get a message and traceback when your application tries to read an uninitialized scalar variable, if that is the problem. Fix, and repeat.
You may also want to make sure you don't exceed the stack limit with -auto or -openmp, by setting
ulimit -s unlimited

[juansheng]

Thank you for your replies.

Here is the codes snippet of the project. matsym is a subroutine to make a matrix symmetric and matinv to calculate inverse of a matrix using dgetri/dgetrf in lapack library. These two subroutines were defined in mat_tools module.

Here are my tests:

[bash]$ make clean && make # w/o -openmp
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas mat_tools.f90
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas main_pim.f90
ifort -O2 -opt-matmul -fpconstant -llapack -lblas mat_tools.o main_pim.o -o test
$ ./test
  0.0000  0.00000E+00
  0.2000  0.42708E-02
  0.4000  0.14860E-01
  0.6000  0.11926E-01
  0.8000  0.61343E-02
  1.0000  0.25699E-02
$ make clean && make # w/ -openmp
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas -openmp mat_tools.f90
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas -openmp main_pim.f90
ifort -O2 -opt-matmul -fpconstant -llapack -lblas -openmp mat_tools.o main_pim.o -o test
$ ./test
  0.0000  0.00000E+00
  0.2000 -0.78223E-01
  0.4000 -0.21740E+02
  0.6000 -0.12120E+05
  0.8000 -0.74834E+07
  1.0000 -0.48519E+10
$ make clean && make # w/ -auto
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas -auto mat_tools.f90
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas -auto main_pim.f90
ifort -O2 -opt-matmul -fpconstant -llapack -lblas -auto mat_tools.o main_pim.o -o test
$ ./test
 0.0000 0.00000E+00
 0.2000 -0.36805E-01
 0.4000 -0.22996E+02
 0.6000 -0.13986E+05
 0.8000 -0.88747E+07
 1.0000 -0.57955E+10
$ make clean && make # w/ -openmp -check uninit -traceback as suggested by Martyn Corden
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas -openmp -check uninit -traceback mat_tools.f90
ifort -c -O2 -opt-matmul -fpconstant -llapack -lblas -openmp -check uninit -traceback main_pim.f90
ifort -O2 -opt-matmul -fpconstant -llapack -lblas -openmp -check uninit -traceback mat_tools.o main_pim.o -o test
$ ./test
  0.0000  0.00000E+00
  0.2000  0.90829E-02
  0.4000  0.60872E-01
  0.6000 -0.22080E-01
  0.8000 -0.39204E+00
  1.0000  0.75379E+00
$ ifort -v
ifort version 12.1.0[/bash]

Four tries introduce four different results. That's a bit astonishing. Are there any implicit bugs in my codes?

[TimP]

Are you linking against blas and lapack libraries built by gfortran? This seems to confirm the suspicion that you like to live dangerously, as does your apparent reliance on -fpconstant and expectation that someone could diagnose bugs in your unseen source code.

[juansheng]

I recompiled the source codes without any specific options of ifort, using matrix inverse subroutine written by myself. The results from w/ -openmp and w/o -openmp differ exactly as my previous post. Therefore, this problem should not be related with blas and lapack libraries or -fpconstant option of ifort compiler.

I can confirm that the unseen mat_tools module is nothing special, just including matsym and matinv subroutines, which have coorperated well with other projects for a long time.

It's interesting that gfortran with -fopenmp rocks. So I double there might be some bugs in implementation of -openmp of intel fortran compiler.

[TimP]

If gfortran compiled lapack and blas work with ifort, it's a matter of luck, and the possibility that you don't hit run-time library conflicts in your usage. There's little reason to try it, since the MKL lapack and BLAS whcih come with ifort are designed to work with either gfortran or ifort (not both together). Linking a combination of gfortran and ifort against Intel OpenMP library could increase the incidence of run-time conflicts, even though combinations of icc and gcc C and C++ and one or the other Fortran compiler are supported.
You can't expect libgomp to work with ifort, although it does work very well with gfortran, and may outperform libiomp5 in specific cases, due to shortcuts in implementation of explicit barriers.

[pbkenned1]

Tim stated exactly what the issue might be. If the lapack and blas libs you're linking against with ifort were compiled with gfortran, then there is no guarantee at all of correct runtime behaviour. Mixing objects compiled by ifort with objects compiled by gfortran is not supported.

Patrick

[juansheng]

Thanks for your explanation.

I've cancelled the blas/lapack linking operation built by gfortran with ifort, the results were reviewed in my previous post. So would these libraries always affect my issue? It's a bit ridiculous. Sorry if I misunderstand what Tim said.

Anyway, how should I use openmp library with ifort exactly?

[TimP]

Asking ifort to select the OpenMP library (by including the -openmp option) is usually the best way.
There's no inherent problem in having the gfortran lapack and blas libraries installed on your platform, but with ifort you should replace them with MKL.

[juansheng]

Yes, I did that too. Current issue is ifort w/ -openmp and w/o -openmp will generate different results and the former one is totally incorrect. lapack and blas libraries are currently _NOT_ involved in these two conditions.

[bash]$ make clean && make
ifort -c  mat_tools.f90
ifort -c  main_pim.f90
ifort  mat_tools.o main_pim.o -o test
$ ./test 
  0.0000  0.00000E+00
  0.2000  0.42708E-02
  0.4000  0.14860E-01
  0.6000  0.11926E-01
  0.8000  0.61343E-02
  1.0000  0.25699E-02
$ vim Makefile 
$ make clean && make
ifort -c -openmp mat_tools.f90
ifort -c -openmp main_pim.f90
ifort -openmp mat_tools.o main_pim.o -o test
$ ./test 
  0.0000  0.00000E+00
  0.2000 -0.78223E-01
  0.4000 -0.21740E+02
  0.6000 -0.12120E+05
  0.8000 -0.74834E+07
  1.0000 -0.48519E+10
$ [/bash]

This has drived me crazy for a long time for all of my projects. I re-paste the codes HERE. Please check them and give me some suggestions. Thank you!

[pbkenned1]

Thanks for providing the codes, I'll take a look. I assume the output _without_ using -openmp is correct.

Patrick Kennedy

[pbkenned1]

Please attach mat_tools.f90, or whatever file defines module ''mat_tools'. I can't compile what you pasted at the link:

>ifort global_mod.f90

Intel Visual Fortran Intel 64 Compiler XE for applications running on Intel 64, Version 12.1.1.258 Build 20111011

global_mod.f90(18): error #7002: Error in opening the compiled module file. Che

ck INCLUDE paths. [MAT_TOOLS]

use mat_tools, only : matsym

Thanks,
Patrick

[juansheng]

OK, thanks. Here is the mat_tools module. And yes, ifort without -openmp generates the correct results.

Jie

[pbkenned1]

Hello Jie,
Thanks for posting the mat_tools module, that allowed me to reproduce the issue. It looks like a very low-level bug in the compiler, since incorrect outputs are produced at -O0 (with either -auto or -openmp). The only 'workaround' I've found is to compile at -O0 with '-check bounds'. I haven't found any workaround for -O1 or higher.

It will take me some time to reduce your codes so something the developers can fix (current test case is too large to be accepted). If you have a smaller example of the problem, please post that.

I do know that things go bad early in the program, for example, the outputs are wrong at the first write to your data file:

open(8, file='x1.dat')
write(8, "(2E12.4)") T0, Yi(1)

If you don't have a smaller example, then I'll start working backwards from the above to isolate the problem.

Thank you,
Patrick Kennedy

[juansheng]

Thanks for your suggestions. I introduced a new and a bit smaller case for debug. The mat_tools module has been removed completely to avoid coupling with issues of any custom matrix operations. This new project only contains about 100 lines and can be found here.

My results generated from this new case:

[bash]$ ifort debug_omp.f90 
$ ./a.out
  0.0000  0.00000E+00
  0.2500  0.51598E-02
  0.5000  0.40642E-01
  0.7500  0.12390E+00
  1.0000  0.22813E+00
$ ifort debug_omp.f90 -openmp
$ ./a.out
  0.0000  0.00000E+00
  0.2500  0.51585E-02
  0.5000  0.40631E-01
  0.7500  0.12387E+00
  1.0000  0.22808E+00[/bash]

Although the difference is now not so large as previous cases, strictly speaking the results w/ -openmp are always wrong. I've tested the gfortran w/ or w/o -fopenmp, these two results are exactly same as ifort w/o -openmp. HIT.


Jie

[juansheng]

Sorry if duplicate posting since I can't see my new reply here. :)

Thanks for your suggestions. I introduced a new and a bit smaller case for debug. The mat_tools module has beem remove completely to avoid coupling with possible issues of any matrix operations. The new project only contains about 100 lines and can be found here.

Results from this new case:

[bash]$ ifort debug_omp.f90 
$ ./a.out 
  0.0000  0.00000E+00
  0.2500  0.51598E-02
  0.5000  0.40642E-01
  0.7500  0.12390E+00
  1.0000  0.22813E+00
$ ifort debug_omp.f90 -openmp
$ ./a.out 
  0.0000  0.00000E+00
  0.2500  0.51585E-02
  0.5000  0.40631E-01
  0.7500  0.12387E+00
  1.0000  0.22808E+00
[/bash]

Although the differece is now not so large as previous cases, strictly speaking results with -openmp are incorrect. Thank you again.


Jie

[pbkenned1]

Hello Jie,
Thank you for providing a reduced test case. That allowed me to isolate the problem well enough to report the defect to compiler engineering. I'll pass along updates I receive from them in this thread.

Defect tracking number: DPD200177011

Patrick Kennedy

[juansheng]

Hello Patrick Kennedy,

Any news there? Is the intel defect tracking system not open for public? Thank you.


Jie