Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent Output

oacikgoz — Mon, 01 Nov 2010 23:58:55 GMT

Hi everyone,

I'm going nuts with my code, because what is happening is beyond my comprehension. When I compile my code with -g option, or -check all, or -check bounds flag, the code runs perfectly fine, though, as you can guess, very slow.

Whenever I use any optimization flag, the code fails to run, gives random segmentation faults. More importantly, the code has something of this sort:

x(1)=...

x(2)=...

x(64)=...

When I add WRITE(6,*) x(1) at the end, it gives me a number totally different from what I set it to. (gives me zero) x is an allocatable array which is properly allocated to be of size 64.

Here are some technical details:

I do not have Intel C/C++ compiler and therefore OpenMPI was compiled with gcc, g++ and ifort combined, with no special flags. If I use gfortran, the code runs with no problems, regardless of the optimization level.

I also tried to use MPICH2 and still the problem persists. It's not specific to OpenMPI. There is a compiler bug in Intel Compiler, or, there is an incompatibility between GCC 4.2 and ifort.

What do you think is the root of the problem?

Thanks

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Dmitry_K_Intel2 — Tue, 02 Nov 2010 07:22:48 GMT

Hi oacikgoz,

I would not blame Intel compiler in this problem. Could you please download an evaluation copy of the Intel MPI library and give it a try. Download it from www.intel.com/go/mpi.

Pay attention that you need to use mpiifort for Intel Fortran Compiler.

Best wishes,
Dmitry

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

oacikgoz — Tue, 02 Nov 2010 14:36:33 GMT

Thanks for your reply. I cannot install MPI Intel because it's not available on Mac's. I'm using a 12-core Mac Pro with Mac OS X Installed.

Any other suggestions?

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Dmitry_K_Intel2 — Wed, 03 Nov 2010 08:47:01 GMT

oacikgoz,

If you are sure that the issue is related to Intel Fortran compiler you can submit a request via Premier porviding a reproducer.
Unfortunately we cannot help you with Mac OS because we don't even have this OS on our servers.

Best wishes,
Dmitry

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

oacikgoz — Wed, 03 Nov 2010 16:35:12 GMT

I just can't believe what I see. The compiler screws up for some reason for the following code whenever I use default optimization. When I turn off optimization it works just fine... Very odd. The screen output, when I run it on 4 nodes is,

-0.999305041735772 -0.999305041735772

-1.770442141520288E-003 -0.999305041735772

-0.999305041735772 -0.999305041735772

-1.770442141520288E-003 -0.999305041735772

-0.999305041735772 -0.999305041735772

-1.770442141520288E-003 -0.999305041735772

-0.999305041735772 -0.999305041735772

-1.770442141520288E-003 -0.999305041735772

The correct output should have been

-0.999305041735772 1.783280721696433E-003

3.474791321139148E-004 1.783280721696433E-003

-0.999305041735772 1.783280721696433E-003

3.474791321139148E-004 1.783280721696433E-003

-0.999305041735772 1.783280721696433E-003

3.474791321139148E-004 1.783280721696433E-003

-0.999305041735772 1.783280721696433E-003

3.474791321139148E-004 1.783280721696433E-003

PROGRAM main

IMPLICIT NONE

INCLUDE 'mpif.h'

INTEGER :: rank

INTEGER :: nproc

INTEGER :: ierror

INTEGER :: i

INTEGER :: status(MPI_STATUS_SIZE)

REAL(8), DIMENSION(:), ALLOCATABLE :: xt

REAL(8), DIMENSION(:), ALLOCATABLE :: wt

REAL(8), DIMENSION(:), ALLOCATABLE :: outp

CALL MPI_INIT(ierror)

CALL MPI_COMM_RANK(MPI_COMM_WORLD,rank,ierror)

CALL MPI_COMM_SIZE(MPI_COMM_WORLD,nproc,ierror)

DO i=1,nproc

IF(rank==i-1) CALL preparevariables()

END DO

CALL MPI_FINALIZE(ierror)

CONTAINS

SUBROUTINE preparevariables()

IMPLICIT NONE

ALLOCATE(xt(64))

ALLOCATE(wt(64))

ALLOCATE(outp(64))

xt(1) = - 0.999305041735772139456905624346D+00

xt(2) = - 0.996340116771955279346924500676D+00

xt(3) = - 0.991013371476744320739382383443D+00

xt(4) = - 0.983336253884625956931299302157D+00

xt(5) = - 0.973326827789910963741853507352D+00

xt(6) = - 0.961008799652053718918614121897D+00

xt(7) = - 0.946411374858402816062481491347D+00

xt(8) = - 0.929569172131939575821490154559D+00

xt(9) = - 0.910522137078502805756380668008D+00

xt(10) = - 0.889315445995114105853404038273D+00

xt(11) = - 0.865999398154092819760783385070D+00

xt(12) = - 0.840629296252580362751691544696D+00

xt(13) = - 0.813265315122797559741923338086D+00

xt(14) = - 0.783972358943341407610220525214D+00

xt(15) = - 0.752819907260531896611863774886D+00

xt(16) = - 0.719881850171610826848940217832D+00

xt(17) = - 0.685236313054233242563558371031D+00

xt(18) = - 0.648965471254657339857761231993D+00

xt(19) = - 0.611155355172393250248852971019D+00

xt(20) = - 0.571895646202634034283878116659D+00

xt(21) = - 0.531279464019894545658013903544D+00

xt(22) = - 0.489403145707052957478526307022D+00

xt(23) = - 0.446366017253464087984947714759D+00

xt(24) = - 0.402270157963991603695766771260D+00

xt(25) = - 0.357220158337668115950442615046D+00

xt(26) = - 0.311322871990210956157512698560D+00

xt(27) = - 0.264687162208767416373964172510D+00

xt(28) = - 0.217423643740007084149648748989D+00

xt(29) = - 0.169644420423992818037313629748D+00

xt(30) = - 0.121462819296120554470376463492D+00

xt(31) = - 0.729931217877990394495429419403D-01

xt(32) = - 0.243502926634244325089558428537D-01

xt(33) = 0.243502926634244325089558428537D-01

xt(34) = 0.729931217877990394495429419403D-01

xt(35) = 0.121462819296120554470376463492D+00

xt(36) = 0.169644420423992818037313629748D+00

xt(37) = 0.217423643740007084149648748989D+00

xt(38) = 0.264687162208767416373964172510D+00

xt(39) = 0.311322871990210956157512698560D+00

xt(40) = 0.357220158337668115950442615046D+00

xt(41) = 0.402270157963991603695766771260D+00

xt(42) = 0.446366017253464087984947714759D+00

xt(43) = 0.489403145707052957478526307022D+00

xt(44) = 0.531279464019894545658013903544D+00

xt(45) = 0.571895646202634034283878116659D+00

xt(46) = 0.611155355172393250248852971019D+00

xt(47) = 0.648965471254657339857761231993D+00

xt(48) = 0.685236313054233242563558371031D+00

xt(49) = 0.719881850171610826848940217832D+00

xt(50) = 0.752819907260531896611863774886D+00

xt(51) = 0.783972358943341407610220525214D+00

xt(52) = 0.813265315122797559741923338086D+00

xt(53) = 0.840629296252580362751691544696D+00

xt(54) = 0.865999398154092819760783385070D+00

xt(55) = 0.889315445995114105853404038273D+00

xt(56) = 0.910522137078502805756380668008D+00

xt(57) = 0.929569172131939575821490154559D+00

xt(58) = 0.946411374858402816062481491347D+00

xt(59) = 0.961008799652053718918614121897D+00

xt(60) = 0.973326827789910963741853507352D+00

xt(61) = 0.983336253884625956931299302157D+00

xt(62) = 0.991013371476744320739382383443D+00

xt(63) = 0.996340116771955279346924500676D+00

xt(64) = 0.999305041735772139456905624346D+00

wt(1) = 0.178328072169643294729607914497D-02

wt(2) = 0.414703326056246763528753572855D-02

wt(3) = 0.650445796897836285611736039998D-02

wt(4) = 0.884675982636394772303091465973D-02

wt(5) = 0.111681394601311288185904930192D-01

wt(6) = 0.134630478967186425980607666860D-01

wt(7) = 0.157260304760247193219659952975D-01

wt(8) = 0.179517157756973430850453020011D-01

wt(9) = 0.201348231535302093723403167285D-01

wt(10) = 0.222701738083832541592983303842D-01

wt(11) = 0.243527025687108733381775504091D-01

wt(12) = 0.263774697150546586716917926252D-01

wt(13) = 0.283396726142594832275113052002D-01

wt(14) = 0.302346570724024788679740598195D-01

wt(15) = 0.320579283548515535854675043479D-01

wt(16) = 0.338051618371416093915654821107D-01

wt(17) = 0.354722132568823838106931467152D-01

wt(18) = 0.370551285402400460404151018096D-01

wt(19) = 0.385501531786156291289624969468D-01

wt(20) = 0.399537411327203413866569261283D-01

wt(21) = 0.412625632426235286101562974736D-01

wt(22) = 0.424735151236535890073397679088D-01

wt(23) = 0.435837245293234533768278609737D-01

wt(24) = 0.445905581637565630601347100309D-01

wt(25) = 0.454916279274181444797709969713D-01

wt(26) = 0.462847965813144172959532492323D-01

wt(27) = 0.469681828162100173253262857546D-01

wt(28) = 0.475401657148303086622822069442D-01

wt(29) = 0.479993885964583077281261798713D-01

wt(30) = 0.483447622348029571697695271580D-01

wt(31) = 0.485754674415034269347990667840D-01

wt(32) = 0.486909570091397203833653907347D-01

wt(33) = 0.486909570091397203833653907347D-01

wt(34) = 0.485754674415034269347990667840D-01

wt(35) = 0.483447622348029571697695271580D-01

wt(36) = 0.479993885964583077281261798713D-01

wt(37) = 0.475401657148303086622822069442D-01

wt(38) = 0.469681828162100173253262857546D-01

wt(39) = 0.462847965813144172959532492323D-01

wt(40) = 0.454916279274181444797709969713D-01

wt(41) = 0.445905581637565630601347100309D-01

wt(42) = 0.435837245293234533768278609737D-01

wt(43) = 0.424735151236535890073397679088D-01

wt(44) = 0.412625632426235286101562974736D-01

wt(45) = 0.399537411327203413866569261283D-01

wt(46) = 0.385501531786156291289624969468D-01

wt(47) = 0.370551285402400460404151018096D-01

wt(48) = 0.354722132568823838106931467152D-01

wt(49) = 0.338051618371416093915654821107D-01

wt(50) = 0.320579283548515535854675043479D-01

wt(51) = 0.302346570724024788679740598195D-01

wt(52) = 0.283396726142594832275113052002D-01

wt(53) = 0.263774697150546586716917926252D-01

wt(54) = 0.243527025687108733381775504091D-01

wt(55) = 0.222701738083832541592983303842D-01

wt(56) = 0.201348231535302093723403167285D-01

wt(57) = 0.179517157756973430850453020011D-01

wt(58) = 0.157260304760247193219659952975D-01

wt(59) = 0.134630478967186425980607666860D-01

wt(60) = 0.111681394601311288185904930192D-01

wt(61) = 0.884675982636394772303091465973D-02

wt(62) = 0.650445796897836285611736039998D-02

wt(63) = 0.414703326056246763528753572855D-02

wt(64) = 0.178328072169643294729607914497D-02

WRITE(*,*) xt(1),wt(1)

xt=(xt+1.0D0)*0.5D0

WRITE(*,*) xt(1),wt(1)

END SUBROUTINE preparevariables

END PROGRAM main

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

oacikgoz — Wed, 03 Nov 2010 16:41:04 GMT

One more thing, I just realized that this has nothing to do with MPI. Commenting out MPI related lines also reproduces the same error...

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Dmitry_K_Intel2 — Sat, 06 Nov 2010 09:47:28 GMT

The answer is here: http://software.intel.com/en-us/forums/showthread.php?t=78427

topic Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent in Intel® MPI Library

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent Output

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent

Intel Fortran Compiler 11.1 and OpenMPI 1.4 bug - Inconsistent