Serious optimizer bug in current Intel Fortran compilers

[mecej4]

There is an insidious optimizer bug in Intel Fortran compilers, version 8 and later, including the current ones.
The bug is sensitive in that moving lines of code with no semantic change or inserting WRITE statements to hunt for the bug makes the bug disappear. It is next to impossible to hunt for this bug with a symbolic disassembler.

Many hours of work were needed to pare down source code from a rather large package to a size small enough to report here and yet large enough to replicate the bug. The example given here has no array overruns or usage of uninitialized variables. The line of code that the optimizer erroneously skips is marked in the example source code. Moving that line to a different location can make the bug go away, but this happens in an unpredictable way.

Here are the results from various machines. In all cases, running with optimization disabled gives correct results. On some machines, even -O2 and -fast give correct results when -Ot or -O1 get hit by the bug.

Machine: Linux IA64 2.6.18-164.10.1.el5

Compilers:

Intel Fortran IA-64 Compiler Version 10.1 Build 20080801 Package ID: l_fc_p_10.1.018

and

Intel Fortran IA-64 Compiler Version 10.0 Build 20070809 Package ID: l_fc_c_10.0.026

-C nvar = 11, ISTAJAC = 1 4 7 8 correct

-O0 nvar = 11, ISTAJAC = 1 4 7 8 correct

-O1 nvar = 11, ISTAJAC = 4 4 6 7 wrong

-O2 nvar = 11, ISTAJAC = 4 4 5 8 wrong

-fast nvar = 11, ISTAJAC = 4 4 5 8 wrong

Machine: Linux x86_64 2.6.27.45-0.1-default

Compiler:

Intel Fortran Intel 64 Compiler Version 11.1 Build 20090827 Package ID: l_cprof_p_11.1.056

-C nvar = 11, ISTAJAC = 1 4 7 8 correct

-O0 nvar = 11, ISTAJAC = 1 4 7 8 correct

-O1 nvar = 11, ISTAJAC = 1 3 6 7 wrong

-O2 nvar = 11, ISTAJAC = 1 4 7 8 correct

-fast nvar = 11, ISTAJAC = 1 4 7 8 correct

Machine:

Windows7 X64 Home Premium

Compiler:

Intel Visual Fortran Compiler Version 11.0 (Intel64) Build 220090318 Package ID: w_cprof_p_11.0.074

and

Intel Visual Fortran Compiler Version 11.0 (IA32) Build 220090318 Package ID: w_cprof_p_11.0.074

-C nvar = 11, ISTAJAC = 1 4 7 8 correct

-Od nvar = 11, ISTAJAC = 1 4 7 8 correct

-Ot nvar = 11, ISTAJAC = 1 1 4 8 wrong

-O2 nvar = 11, ISTAJAC = 1 1 4 8 wrong

-fast nvar = 11, ISTAJAC = 1 1 4 8 wrong

_______________________________________________________

[fortran]program IOPTBUG
   
   implicit none
   integer :: nvarg
   integer, allocatable :: istaj( : )
   
   call initw(nvarg,istaj)
   
   write(*,10)nvarg, istaj
   stop
   
10 format(' nvar = ',I2,',  ISTAJAC  = ',4I4)


     CONTAINS

       SUBROUTINE INITW(nvargp, ISTAJC )

       integer, parameter :: n = 3, ng = 6, nel = 3, ntotel = 3, nvrels = 7, nnza = 4, numvar = 3

       INTEGER, ALLOCATABLE, DIMENSION( : ) :: ISTAJC

       INTEGER, INTENT( OUT ) :: nvargp

       LOGICAL :: skipg = .false.
       INTEGER, DIMENSION( ntotel  ) :: IELING = (/ 1, 2, 3 /)
       INTEGER, DIMENSION( nvrels )  :: ISTADG = (/ 1, 1, 2, 3, 3, 4, 4 /)
       INTEGER, DIMENSION( ng  + 1 ) :: ISTADA = (/ 1, 2, 2, 2, 3, 3, 5 /)
       INTEGER, DIMENSION( nel + 1 ) :: ISTAEV = (/ 1, 4, 6, 8 /)
       INTEGER, DIMENSION( nvrels  ) :: IELVAR = (/ 2, 1, 3, 2, 3, 1, 2 /)
       INTEGER, DIMENSION( nnza    ) :: ICNA   = (/ 1, 2, 1, 2 /)

       LOGICAL, DIMENSION( 6  ) :: GXEQX = (/.false.,.true.,.false.,.true.,.false.,.false./)
       INTEGER, DIMENSION( 6 )  :: KNDOFG = (/1,1,1,1,1,2/)
       INTEGER, DIMENSION( 7 )  :: ISTAGV = (/ 1,2,5,7,8,10,12 /)
       INTEGER, DIMENSION( 11 ) :: ISVGRP
       INTEGER, DIMENSION( 3 )  :: ISWKSP

       INTEGER :: i , j , k , L , ig
       INTEGER :: iel

       ALLOCATE( ISTAJC( n + 1 ) )

       nvargp = 0

       ISWKSP( : numvar ) = 0

       DO ig = 1, ng
         IF ( skipg ) THEN ; IF ( KNDOFG( ig ) == 0 ) CYCLE ; END IF
         DO k = ISTADG( ig ), ISTADG( ig + 1 ) - 1
           iel = IELING( k )
           DO j = ISTAEV( iel ), ISTAEV( iel + 1 ) - 1
             i = IELVAR( j )
             IF ( ISWKSP( i ) < ig ) THEN
               ISWKSP( i ) = ig
               nvargp = nvargp + 1
             END IF
           END DO
         END DO

         DO j = ISTADA( ig ), ISTADA( ig + 1 ) - 1
           i = ICNA( j )
           IF ( i <= numvar ) THEN
             IF ( ISWKSP( i ) < ig ) THEN
                ISWKSP( i ) = ig
                nvargp = nvargp + 1
             END IF
           END IF
         END DO
       END DO

       ISTAJC( : n + 1 ) = 0
       ISWKSP( : numvar ) = 0

       nvargp = 0

       DO ig = 1, ng

         DO k = ISTADG( ig ), ISTADG( ig + 1 ) - 1
           iel = IELING( k )
           DO j = ISTAEV( iel ), ISTAEV( iel + 1 ) - 1
             i = IELVAR( j )
             IF ( ISWKSP( i ) < ig ) THEN
               ISWKSP( i ) = ig
               nvargp = nvargp + 1
               ISVGRP( nvargp ) = i
             END IF
           END DO
         END DO
         DO j = ISTADA( ig ), ISTADA( ig + 1 ) - 1
           i = ICNA( j )
           IF ( i <= numvar ) THEN
             IF ( ISWKSP( i ) < ig ) THEN
               ISWKSP( i ) = ig
               nvargp = nvargp + 1
               ISVGRP( nvargp ) = i
             END IF
           END IF
         END DO
         IF ( .NOT. GXEQX( ig ) ) THEN
           DO j = ISTAGV( ig ), nvargp
             L = ISVGRP( j ) + 1
             ISTAJC( L ) = ISTAJC( L ) + 1
           END DO
         END IF
       END DO
       ISWKSP( : n ) = 0
       ISTAJC( 1 ) = 1                   ! This is the statement that the optimizer skips
       DO i = 2, n + 1
         ISTAJC( i ) = ISTAJC( i ) + ISTAJC( i - 1 )
       END DO
       DO ig = 1, ng
         IF ( skipg ) THEN ; IF ( KNDOFG( ig ) == 0 ) CYCLE ; END IF
         IF ( .NOT. GXEQX( ig ) ) THEN
           DO i = ISTAGV( ig ), ISTAGV( ig + 1 ) - 1
             L = ISVGRP( i )
             j = ISTAJC( L )
             ISTAJC( L ) = j + 1
           END DO
         END IF
       END DO
       DO i = n, 2, - 1
         ISTAJC( i ) = ISTAJC( i - 1 )
       END DO
       ISTAJC( 1 ) = 1
       return
       END SUBROUTINE INITW

END Program IOPTBUG[/fortran]

[Xiaoping_D_Intel]

Thanks for reporting the problem. I am trying to reproduce it and will give you an update soon.

[Xiaoping_D_Intel]

The problem has been reproduced and forwarded to developers for fixing. I will post update on it here.

[mecej4]

Thank you.

In case that it may be of help, here is an even shorter example that shows the optimization bug with earlier versions (but NOT V11.x) of the Intel Fortran compiler.

With Version 10 on Linux IA64, the results are:

ifort -O0 1 1 1 1
ifort -O1 1 0 0 0
ifort -O2 1 0 0 0

[bash]program IOPTBUG

   implicit none
   integer, allocatable :: istaj( : )

   call initw(istaj)

   write(*,10) istaj
   stop

10 format('  ISTAJAC  = ',4I4)


     CONTAINS

       SUBROUTINE INITW(ISTAJC )

       integer, parameter :: n = 3, ng = 6
       INTEGER, ALLOCATABLE, DIMENSION( : ) :: ISTAJC
       INTEGER :: i

       ALLOCATE( ISTAJC( n + 1 ) )

       ISTAJC( : n + 1 ) = 0
       ISTAJC( 1 ) = 1                   ! This is the statement that the optimizer skips

       DO i = 2, n + 1
         ISTAJC( i ) = ISTAJC( i ) + ISTAJC( i - 1 )
       END DO

       DO i = n, 2, - 1
         ISTAJC( i ) = ISTAJC( i - 1 )
       END DO
       ISTAJC( 1 ) = 1
       return

       END SUBROUTINE INITW

END Program IOPTBUG
[/bash]

[Xiaoping_D_Intel]

Thanks for the simpler test case; however, it can not expose the runtime problem in the newversion compiler sojust let developers use the original one.

[woshiwuxin]

For this concise code, I also observed the peculiar output on both Mac and Linux x86_64 with optimization option of -O1 with Intel Fortran Compiler 11.1. However, a good news is that the -O2 option, which is the default optimization option, produces correct output. :D

Quoting mecej4

Thank you.

In case that it may be of help, here is an even shorter example that shows the optimization bug with earlier versions (but NOT V11.x) of the Intel Fortran compiler.

With Version 10 on Linux IA64, the results are:

ifort -O0 1 1 1 1
ifort -O1 1 0 0 0
ifort -O2 1 0 0 0

[mecej4]

That's "good news", really!

As the longer example establishes, your experience merely shows that the optimizer bug knows how to hide well, only to rise up and strike when you least expect it, deep inside your 200,000 lines-of-code program and only on certain days.

Until this bug is fixed, I'd not put faith in any results that cannot be reproduced (exactly, if using only integer types, and within the limitations of floating point arithmetic, otherwise) with -O0 or with another Fortran compiler. Knowing that this bug exists is similar to having a car whose engine is prone to stall at speeds above 70 MPH (110 kMPH).

[Xiaoping_D_Intel]

The probem has been fixed and will be included in a future update.

[mecej4]

The rapidity with which this fix has been implemented is impressive.

We look forward to the next issue of the compiler with this fix incorporated.

[Xiaoping_D_Intel]

The bug has been fixed in the latest 11.1.073 update.

You may download this compiler package from our Registration Center: https://registrationcenter.intel.com/regcenter/register.aspx Please login with yourpremier Support Login ID and Password, select 'Product Downloads'.

Thanks,
Xiaoping

[mecej4]

Thanks. I verified that the bug has been fixed by running the new release of the compiler on a number of test cases where errors had occurred with the older releases of the compiler.

My compliments to the developers!

[intelFortranUser]

We would like to take advantage of this important fix for our ifort 11.1 based product.
Unfortuantely for windows the latest version of the compiler (update 9) is 11.1.072.
How can we get hold of 11.1.073 or newer 11.1 version for Windows?

thanks
Wolfgang

[Kevin_D_Intel]

The Linux 11.1.072 id maps to the Intel Compiler Professional Edition 11.1 "Update 7". The Windows equivalent id from Update 7 is 11.0.067 (see here for a mapping).

Refer to How do I get an older version of an Intel Software Development Product? for details on obtaining the older versions.

[intelFortranUser]

thanks for the clarification!