Illegal instruction when running programs compiled by mpifort

[Haoyan_H_]

Hi all,

Recently I compiled VASP using Intel Fortran Compilers, but the program does not seem to work properly on my cluster. I got errors like:

 running on    1 total cores
 distrk:  each k-point on    1 cores,    1 groups
 distr:  one band on    1 cores,    1 groups
 using from now: INCAR     
 vasp.5.3.2 13Sep12 (build Dec 31 2015 22:13:40) complex                        
  
forrtl: severe (168): Program Exception - illegal instruction
Image              PC                Routine            Line        Source             
vasp               00000000011296A5  Unknown               Unknown  Unknown
vasp               0000000001127467  Unknown               Unknown  Unknown
vasp               00000000010BD1F4  Unknown               Unknown  Unknown
vasp               00000000010BD006  Unknown               Unknown  Unknown
vasp               000000000105F2D6  Unknown               Unknown  Unknown
vasp               00000000010655A8  Unknown               Unknown  Unknown
libpthread.so.0    00007F7374785340  Unknown               Unknown  Unknown
vasp               0000000000494457  Unknown               Unknown  Unknown
vasp               0000000000495AEC  Unknown               Unknown  Unknown
vasp               0000000000406468  Unknown               Unknown  Unknown
vasp               000000000040531E  Unknown               Unknown  Unknown
libc.so.6          00007F73740CBEC5  Unknown               Unknown  Unknown
vasp               0000000000405229  Unknown               Unknown  Unknown

The disassembly of the code is:

(gdb) disassemble 
Dump of assembler code for function lattice_mp_lattic_:
   0x0000000000494420 <+0>:    vmovsd 0x30(%rdi),%xmm0
   0x0000000000494425 <+5>:    vmovsd 0x40(%rdi),%xmm15
   0x000000000049442a <+10>:    vmovsd 0x38(%rdi),%xmm14
   0x000000000049442f <+15>:    vmovsd 0x18(%rdi),%xmm8
   0x0000000000494434 <+20>:    vmulsd %xmm15,%xmm0,%xmm3
   0x0000000000494439 <+25>:    vmulsd %xmm14,%xmm8,%xmm10
   0x000000000049443e <+30>:    vmovsd 0x28(%rdi),%xmm2
   0x0000000000494443 <+35>:    vmovsd 0x48(%rdi),%xmm13
   0x0000000000494448 <+40>:    vmovsd 0x20(%rdi),%xmm1
   0x000000000049444d <+45>:    vmovsd 0x10(%rdi),%xmm6
   0x0000000000494452 <+50>:    vmovsd 0x8(%rdi),%xmm7
=> 0x0000000000494457 <+55>:    vfmsub231sd %xmm2,%xmm13,%xmm3
   0x000000000049445c <+60>:    vmulsd %xmm1,%xmm13,%xmm4
   0x0000000000494460 <+64>:    vfmsub231sd %xmm13,%xmm7,%xmm10
   0x0000000000494465 <+69>:    vmulsd %xmm6,%xmm13,%xmm9
   0x0000000000494469 <+73>:    vmulsd %xmm7,%xmm15,%xmm11
   0x000000000049446d <+77>:    vfmsub231sd %xmm0,%xmm14,%xmm4
   0x0000000000494472 <+82>:    vmulsd %xmm2,%xmm8,%xmm12
   0x0000000000494476 <+86>:    vfmsub231sd %xmm15,%xmm8,%xmm9
   0x000000000049447b <+91>:    vmulsd %xmm13,%xmm13,%xmm13
   0x0000000000494480 <+96>:    vfmsub231sd %xmm6,%xmm14,%xmm11
   0x0000000000494485 <+101>:    vmulsd %xmm14,%xmm2,%xmm5
   0x000000000049448a <+106>:    vfmsub231sd %xmm6,%xmm0,%xmm12
   0x000000000049448f <+111>:    vfmadd213sd %xmm13,%xmm14,%xmm14
   0x0000000000494494 <+116>:    vmulsd %xmm7,%xmm0,%xmm13
   0x0000000000494498 <+120>:    vfmsub231sd %xmm15,%xmm1,%xmm5
   0x000000000049449d <+125>:    vmulsd %xmm0,%xmm0,%xmm0
   0x00000000004944a1 <+129>:    vfmadd213sd %xmm14,%xmm15,%xmm15
   0x00000000004944a6 <+134>:    vfmsub231sd %xmm8,%xmm1,%xmm13
   0x00000000004944ab <+139>:    vmulsd %xmm1,%xmm6,%xmm14
   0x00000000004944af <+143>:    vfmadd213sd %xmm0,%xmm1,%xmm1
   0x00000000004944b4 <+148>:    vsqrtsd %xmm15,%xmm15,%xmm15
   0x00000000004944b9 <+153>:    vfmsub231sd %xmm2,%xmm7,%xmm14
   0x00000000004944be <+158>:    vfmadd213sd %xmm1,%xmm2,%xmm2

I looked up my /proc/cpuinfo, and the flags are:

fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid

seem to have SSE support. I'd like to know if anything is wrong with my setup. Thanks!

[Steven_L_Intel1]

That's an FMA instruction, available only in fairly recent processors. Which compiler options did you use when compiling? Are all your cluster nodes using the same processor model?

My guess is that you used a -m option that wasn't the lowest-common-denominator for your cluster.

[JVanB]

You need avx2, not SSE to run that binary.

[Haoyan_H_]

Steve Lionel (Intel) wrote:

That's an FMA instruction, available only in fairly recent processors. Which compiler options did you use when compiling? Are all your cluster nodes using the same processor model?

My guess is that you used a -m option that wasn't the lowest-common-denominator for your cluster.

That's the reason! I followed some instructions and accidentally set the compiler options "-xCORE_AVX2":

fpp -f_com=no -free -w0 optics.F optics.f90  -DMPI -DHOST=\"LinuxIFC\" -DIFC -DCACHE_SIZE=32000 -DPGF90 -Davoidalloc -DNGZhalf -DMPI_BLOCK=64000 -Duse_collective -DscaLAPACK -DMKL_ILP64 -g -traceback
mpiifort -FR -names lowercase -assume byterecl -I/opt/intel/mkl/include/fftw -O3 -xCORE-AVX2  -I/opt/intel/mkl/include/fftw -c optics.f90

Removing this option solved the problem. Thank you very much!

[Haoyan_H_]

Repeat Offender wrote:

You need avx2, not SSE to run that binary.

Thanks! Removing avx2 compiler option solved the problem.

[Steven_L_Intel1]

Interesting - I would have expected the use of -x to give you a descriptive error when the program started.  You could use -xSSE4.2 and it should work, with better performance.

[Haoyan_H_]

Steve Lionel (Intel) wrote:

Interesting - I would have expected the use of -x to give you a descriptive error when the program started.  You could use -xSSE4.2 and it should work, with better performance.

I don't know what kind of error should the program give, but it just crashed. I have to gdb into the program and find out which instruction caused the crash.

Now I'm using -xAVX. I think it will give better performance.