xe14=>xe15 cause a SIGSEV (openmp)

[Guillaume_d_]

When updating from ifort 14 to ifort 15 (15.0.2 to be precise), several bugs have appear in our programs (while it ran smoothly with ifort 14). One of them seems related to openMP : 

forrtl: severe (174): SIGSEGV, segmentation fault occurred

Image PC Routine Line Source

libintlc.so.5 00007FF35F0F3961 Unknown Unknown Unknown

libintlc.so.5 00007FF35F0F20B7 Unknown Unknown Unknown

libhsall.so 00007FF362794692 Unknown Unknown Unknown

libhsall.so 00007FF3627944E6 Unknown Unknown Unknown

libhsall.so 00007FF36275518C Unknown Unknown Unknown

libhsall.so 00007FF36275BA58 Unknown Unknown Unknown

libpthread.so.0 00007FF35EEDA0A0 Unknown Unknown Unknown

libhsall.so 00007FF3627AA9C3 Unknown Unknown Unknown

libhsall.so 00007FF36277B74B Unknown Unknown Unknown

libhsall.so 00007FF36277ADBC Unknown Unknown Unknown

hspec 000000000042093F hspeccal_ 104 hspeccal.f90

libiomp5.so 00007FF360F7EBB3 Unknown Unknown Unknown

libiomp5.so 00007FF360F53617 Unknown Unknown Unknown

The source code at the location pointed by the traceback looks like :

------------------------------------------------------

character(5) :: css

integer :: inss

!$OMP PARALLEL DO PRIVATE( inss, css)   FIRSTPRIVATE(spectrum)
   do inss=1, nss
      write(css, '(i5.5)') inss

     [...]

end do

------------------------------------------------------

The error does not arises if the number of openmp threads is set to 1. Additionally, the error does not occur on windows (also with ifort 15).

Any suggestions to solve the issue ?

Regards,

  Guillaume

[Steven_L_Intel1]

You will need to provide us with a complete test case we can build and run in order to help you.

[jimdempseyatthecove]

Does the failing system have a large number of hardware threads, and you've done something silly like giving them all unlimited stack?

Errors can also be induced when the code generated requires alignment and the data used is not aligned. If running in Debug build (no optimizations) works, then I suspect you have a data alignment issue.

Jim Dempsey

[Guillaume_d_]

Indeed, the bug does not appear when the number of thread is small (with 1 ot 2 threads the program runs). With 4 or 8 threads it crashes. The crash is "random" ( does not appear all the time, approximatly 1 time every 2 identical runs ).   Code compiled in "debug" mode crashes as often as the optimized one.

Also I have just found a way to to bypass the bug, but I do not really see how it is different from the original code.

!$OMP PARALLEL DO PRIVATE( inss, css,spectrum2nd)   FIRSTPRIVATE(spectrum)
   do inss=1, nss
      write(css, '(i5.5)') inss
     [...]
     call spectrum2nd%write(TRIM(PJName)//"_spec/Response_spectrum_2nd/"//TRIM(qtf_(iqtf)%name)//"_"//css//"_spec.dat", append = .false.)
     [...]
end do

is now replaced by :

!$OMP PARALLEL DO PRIVATE( inss, filename,spectrum2nd)   FIRSTPRIVATE(spectrum)
   do inss=1, nss
          [...]
         write(filename, '(a,i5.5,a)') TRIM(PJName)//"_spec/Response_spectrum_2nd/"//TRIM(qtf_(iqtf)%name)//"_", inss ,"_spec.dat"
         call spectrum2nd%write(TRIM(filename), append = .false.)
          [...]
end do
!$OMP END PARALLEL DO 

[jimdempseyatthecove]

Those are valuable clues. The crash in Debug build indicates the crash is not likely due to unaligned vectors. The lack of, and reduction of, crash with reduced thread count generally indicates a race condition (more threads == higher contentions).

If race conditions do not stand out when visually walking through your program, then you can perform a quazi-binary search by blocking out sections of code in critical section.

!$OMP PARALLEL DO PRIVATE( inss, css,spectrum2nd)   FIRSTPRIVATE(spectrum)
   do inss=1, nss
!$OMP CRITICAL(CHRASH_CHECK)
      write(css, '(i5.5)') inss
     [...]
     call spectrum2nd%write(TRIM(PJName)//"_spec/Response_spectrum_2nd/"//TRIM(qtf_(iqtf)%name)//"_"//css//"_spec.dat", append = .false.)
     [...]
!$OMP END CRITICAL(CHRASH_CHECK)
end do

then move the critical section (either top or bottom)

!$OMP PARALLEL DO PRIVATE( inss, css,spectrum2nd)   FIRSTPRIVATE(spectrum)
   do inss=1, nss
      write(css, '(i5.5)') inss
     [...]
!$OMP CRITICAL(CHRASH_CHECK)
     call spectrum2nd%write(TRIM(PJName)//"_spec/Response_spectrum_2nd/"//TRIM(qtf_(iqtf)%name)//"_"//css//"_spec.dat", append = .false.)
     [...]
!$OMP END CRITICAL(CHRASH_CHECK)
end do

Jim Dempsey