Mysterious ifort compilation failure when -fpp is used with input file is on a NFS volume

[a_zhaogtisoft_com]

This is a bit hard to describe, and unbelievable, too!

All of a sudden, one of our files can not be compiled by ifort anymore, and as I look into the cause of the failure, it looks like -fpp really mess up the compilation - this particular file has a lot of #ifndef, #ifdef blocks.

here is the findings:

1) by using -save-temps, I was able to get the .i file, and with that file, I can compile it successfully with either -fpp or without -fpp option.

2) If the source file is located on a local disk, say, /local/user/builder/source/v740/s/gtsuite/src/mainb.F, the ifort can compile it with -fpp.

3) If the source file is referred on a NFS mount (either direct nfs mount, or automount - auto.home or auto.direct), example: /home/builder/source/v740/s/gtsuite/src/mainb.F, it complains about syntax error - totally non-sense output

-bash-3.2$ /opt/intel/composer_xe_2011_sp1.13.367/bin/ia32/ifort  -extend_source 132 -r8 -DGTDOUBLE -D_ALL_UNIX_ -D_HAVE_FOAM -assume underscore -assume protect_parens -fPIC -align dcommon -fpp -save-temps -module . -I/home/builder/source/v740/s -I. -c -O2 /mnt/source/v740/s/gtsuite/src/mainb.F -o GTsuite/obj/mainb.o  /mnt/source/v740/s/gtsuite/src/mainb.F(1745): error #5149: Illegal character in statement label field  
.        if(idrvopt.eq.0)then
^
/mnt/source/v740/s/gtsuite/src/mainb.F(1745): error #5149: Illegal character in statement label field  [/]
.        if(idrvopt.eq.0)then
-^
/mnt/source/v740/s/gtsuite/src/mainb.F(1745): error #5149: Illegal character in statement label field  
.        if(idrvopt.eq.0)then
----^
/mnt/source/v740/s/gtsuite/src/mainb.F(1745): error #5276: Unbalanced parentheses
.        if(idrvopt.eq.0)then
------------------------^
/mnt/source/v740/s/gtsuite/src/mainb.F(1745): error #5082: Syntax error, found IDENTIFIER 'EXTRA_OUTVOPT' when expecting one of: <END-OF-STATEMENT> ;
.        if(idrvopt.eq.0)then
------^
/mnt/source/v740/s/gtsuite/src/mainb.F(1758): error #6305: An ELSE statement occurred without a corresponding IF THEN statement.
        else
--------^
/mnt/source/v740/s/gtsuite/src/mainb.F(1760): error #6317: An ENDIF occurred without a corresponding IF THEN or ELSE statement.
        endif
--------^
/mnt/source/v740/s/gtsuite/src/mainb.F(1772): error #6309: An ELSEIF statement occurred without a corresponding IF THEN statement.
      elseif(iflag.eq.2)then
------^
/mnt/source/v740/s/gtsuite/src/mainb.F(1795): error #6309: An ELSEIF statement occurred without a corresponding IF THEN statement.
      elseif(iflag.eq.3)then
------^
/mnt/source/v740/s/gtsuite/src/mainb.F(1810): error #6317: An ENDIF occurred without a corresponding IF THEN or ELSE statement.
      endif
------^
/mnt/source/v740/s/gtsuite/src/mainb.F(2682): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
          write(c80,"('Case#',i4,', Step ',i8,', dt = ',1p,g9.3,', t =',g12.5,A)") icase, istep, dt, time, word(:16)
------------------------------------------------------------^
compilation aborted for /mnt/source/v740/s/gtsuite/src/mainb.F (code 1)

Somehow the combination of -fpp with nfs mounted path is consistently giving this failure.

4) I have confirmed the failure with either Solaris 10 as NFS server or RHEL6 as NFS server - so I think the nfs server can not be the cause of this.

5) This problem happened just recently after we added a new nisMap object to our Active Directory, which has nothing to do with the source file location at all. I remember at the same time, we update our Active Directory functional level to use 2008 (from 2000). I could not think of any relationship between the build failure and the above AD changes we made. But the 2 events are correlated in time somehow. 

Has anyone seen stuff like this? Should I open a case with Intel support?

[jimdempseyatthecove]

Do you have any #define's redefining:

if
idrvopt
eq
then

?

Also, check what immediately precedes the errant statement (as it relates to #defines)

Jim Dempsey

[TimP]

Do you have write permission at /tmp/ or where environment variable like TMPDIR may have redirected?

[a_zhaogtisoft_com]

To jimdempseyatthecove,

I do not think we have any #define redirecting. This is a F90 source, we actually do not use any #define anywhere, but we do use some #ifdef or #ifndef or #if defined(....) stuff.

To Tim Prince,

I do have a  /tmp with drwxrwxrwt. There is no TMPDIR in env at all.

[a_zhaogtisoft_com]

To Jim,

The fact that I can first use -save-temps to let -fpp process the .F file, and then pass this intermediate to ifort to have a successful build (location of this intermediate file is not important - it can be local, or on network volume), shows that there is no error in the source files.

[Lorri_M_Intel]

It looks like the intermediate file has an unexpected non-printing character on line 1745.  What is interesting on either side of that line?

The difference between "ifort foo.F" and "ifort -save-temps foo.F" is that the fpp-output file is saved locally with a user-friendly name, rather than to $TEMP with a compiler-generated name.   Actually, fpp is oblivious to the name of its output file, that is all controlled by the "ifort" driver.

                      --Lorri

[a_zhaogtisoft_com]

The line 1745 of the original source actually is empty.

 

#ifndef RT_NO_PLOTS
C--- Beginning of Case
      if(iflag.eq.0)then
C---- this is the line 1745
        if(idrvopt.eq.0)then
          call  foo1(iflag)                   ! acoustics
          if(ichkout.eq.0)then
!            call  foo2(iflag)                   ! mechanical
            call  foo3(iflag)                    ! flow
!            call  foo4(iflag)                   ! thermal
!            call  foo5(iflag)                   ! electrical
            call  foo6(iflag)                    ! control
!            call  foo7(mperiod)                ! compound
!            call  foo8                       ! optimization
!            call  foo9(iflag)                ! Hx plots
          endif
          iopt = 0
        else
          iopt = 1
        endif
... ...

Somehow I think with -fpp, the line # is messed up, it could be referring to the -fpp processed temp file. But even in that -fpp temp file, I do not see anything interesting around line 1745.

It could be interesting that from the original posted errors, one of the error is refering to "EXTRA_OUTVOPT", I do not even anything thing like that in our code.

[a_zhaogtisoft_com]

Another thing, along the line that "EXTRA_OUTVOPT" was not part of any of our code, I found this particular error line could change, for example, right now, it is complaining about
 

/Ydir/savesets/v740/s/gtsuite/src/mainb.F(1745): error #5082: Syntax error, found IDENTIFIER 'RA_OUTIDRVOPT' when expecting one of: <END-OF-STATEMENT> ;

Again, "RA_OUTIDRVOPT" is not in our code. The fact this changes may say something about the nature of the issue. I would blame it on -fpp preprocessor bug.

[jimdempseyatthecove]

I seem to recall a similar issue with a rather old version of FPP (8 or 9). Would by chance, when you setup for the NFS volume, you have also modified PATH, and where it contains a path to an older version of FPP?

Jim Dempsey

[a_zhaogtisoft_com]

I do not think the PATH is modified.

1)

/opt/intel/composer_xe_2011_sp1.13.367/bin/ia32/ifort  -extend_source 132 -r8 -DGTDOUBLE -D_ALL_UNIX_ -D_HAVE_FOAM -assume underscore -assume protect_parens -fPIC -align dcommon -fpp -save-temps -module . -I/home/builder/source/v740/s -I. -c -O2 /mnt/source/v740/s/gtsuite/src/mainb.F -o GTsuite/obj/mainb.o

2)

/opt/intel/composer_xe_2011_sp1.13.367/bin/ia32/ifort  -extend_source 132 -r8 -DGTDOUBLE -D_ALL_UNIX_ -D_HAVE_FOAM -assume underscore -assume protect_parens -fPIC -align dcommon -fpp -save-temps -module . -I/home/builder/source/v740/s -I. -c -O2 /home/builder/source/v740/s/gtsuite/src/mainb.F -o GTsuite/obj/mainb.o 

3)

/opt/intel/composer_xe_2011_sp1.13.367/bin/ia32/ifort  -extend_source 132 -r8 -DGTDOUBLE -D_ALL_UNIX_ -D_HAVE_FOAM -assume underscore -assume protect_parens -fPIC -align dcommon -fpp -save-temps -module . -I/home/builder/source/v740/s -I. -c -O2 /local/users/builder/source/v740/s/gtsuite/src/mainb.F -o GTsuite/obj/mainb.o 

Above three command lines, the actual file is the same file, but 1) is using direct nfs mount, 2) is using an auto.home, and 3) is using the /local/users/builder/ is the local path. Only 3) will get compiled without error. This is from the same bash session.

Also, on the build machine, I only have composer_xe_2011_sp1.13.367, composer_xe_2013_sp1.0.080 installed. The corresponding fpp version is 12.0 and 14.0.x.

[TimP]

You would want to make certain that you don't have both (obsolete) versions of ifort on PATH.  Do you have similar problems with current releases such as the last xe_2013 (not sp1) or the current xe_2013_sp1 (built in January)?  I don't think much priority is given to 2011 problems, nor, evidently, to supporting the initial 2013_sp1 when the 2nd (much improved) update is available and the 3rd may come in 6 weeks or so.

[jimdempseyatthecove]

Assuming PATH not modified by using, for example a different "source ..." between command lines, then this leaves two avenues to explore:

a) Using: /opt/intel/composer_xe_2011_sp1.13.367/bin/ia32/ifort  ... -fpp ...

Is explicitly stating where the ifort is located, but implicitly stating "go find fpp in PATH". What this may introduce into the mix of things is running an older/different version of fpp than the one distributed with the explicitly referenced ifort.

b) The include path for both Fortran INCLUDE and fpp #include, when referencing unspecified include path prefix, always searches the current directory first, then the -I... paths. Depending on if you use either or both include formats this may be an issue. Also, is it a typo that your include folder name is "s"? (-I/home/builder/source/v740/s)

Jim Dempsey

[a_zhaogtisoft_com]

Tim,

I have the same issue (the output is a bit different) with composer_xe_2013_sp1.0.080, the build machine does not have xe_2013 installed, only xe_2013 sp1 (for testing only for now). We have had ton of problems to use ifort v13.x (any version of it).

Jim,

There is no entry of any /opt/intel in PATH setting. So I guess a) is not it. We do have a lot of FORTRAN include that needs -I/home/builder/source/v740/s (s is a real directory name - meaning (s)olver code dir base). But we do not have any #include, because this is 99% f90 codes. We only need -fpp for the C style of #ifdef/#endif for some particular stripped down build when macro is defined.

[Xiaoping_D_Intel]

I have reproduced the error on a NFS system and opened a defect DPD200254618 for it. The error also happens without -fpp option.

[Lorri_M_Intel]

Now that we've had a reproducer in-house, I can handily reproduce the problem, and have a suggested workaround for you until we can fix the actual problem.

At the top of your source file are comment lines containing 130 equal signs.

If you remove one of the equal signs, then the compilation will succeed.

Could you try that, please, in your environment and let us know if it consistently works for you?

               thank you --

                              --Lorri

[a_zhaogtisoft_com]

Lorri,

Removing the '=' does not fix the issue for me.

However, when a developer did a global replacement of tab with space for the file and he also did a dos2unix (I suspect the line ending is not issue, but just would like to report the experience here), the file can be compiled now.

We have found the last change of the source file also co-incite with the failure. A revert to previous version can fix the issue, too. Though when I diff the changes, I really can not tell what could have caused the compiler failure.

About the correlation with NFS mount change, it could be superficial: on one server, all failures happened with NFS path, but on a different server (same OS, and same compiler version/installation), I do find 1 case of failure with a local file. 

At this stage, this looks more like a compiler bug that get triggered under some combination of conditions. But the good news is that we do find a work around by replacing tab (and maybe line ending maybe?).