x64 debug build failure with "fit_lookup: Line_seq_number 0000000000000000 was not found"

[a_zhaogtisoft_com]

I have been struggling with this strange error when building debug version of the our solver using Fortran Compiler XE 13.1.1.171. This issue has already been reported to Intel support,but I have not heard anything from them, and I hope someone here can give me a hint - the issue literately block any of Intel ifort v13.x deployment, if not addressed, we will be stuck to ifort v12.1.

1) Build environment: Fortran Compiler XE 13.1.1.171 hosted on VS2010 (part of the Intel Parallel Studio) on Windows

2) Target: x64 only (with or without -openmp), debug build only

3) Build error:

Compiling with Intel(R) Visual Fortran Compiler XE 13.1.1.171 [Intel(R) 64]... Creating temporary file "RSP1.rsp" with contents [ /nologo /debug:full /Od /fpp /I".." /I"../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\lib\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\chem\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /assume:nosource_include /DDEBUG /reentrancy:threaded /extend_source:132 /real_size:64 /align:rec16byte /align:qcommons /align:sequence /fpe:0 /fpconstant /names:lowercase /iface:cref /assume:underscore /module:"../modules/x64/Double Debug/" /object:"x64\Double Debug/" /traceback /check:bounds /check:uninit /libs:static /threads /dbglibs /c /extfor:f /Qvc10 /Qlocation,link,"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\\bin\amd64" "C:\build\V740\source\cnt\Src\cnt_summ.f" ] Creating command line "ifort @"C:\build\V740\source\cnt\x64\Double Debug\RSP1.rsp"" fit_lookup: Line_seq_number 0000000000000000 was not found fortcom: Fatal: There has been an internal compiler error (C0000005). compilation aborted for C:\build\V740\source\cnt\Src\cnt_summ.f (code 1) Creating temporary file "RSP1.rsp" with contents [ /nologo /debug:full /Od /fpp /I".." /I"../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\lib\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\chem\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /assume:nosource_include /DDEBUG /reentrancy:threaded /extend_source:132 /real_size:64 /align:rec16byte /align:qcommons /align:sequence /fpe:0 /fpconstant /names:lowercase /iface:cref /assume:underscore /module:"../modules/x64/Double Debug/" /object:"x64\Double Debug/" /traceback /check:bounds /check:uninit /libs:static /threads /dbglibs /c /extfor:f /Qvc10 /Qlocation,link,"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\\bin\amd64" "C:\build\V740\source\cnt\Src\cnt_dlay.f" ] Creating command line "ifort @"C:\build\V740\source\cnt\x64\Double Debug\RSP1.rsp"" CNTLIB - 1 error(s), 0 warning(s)

4) Reproducibility:

4.1) For above file, it always fails with the same error output, but only limited to x64 debug build. Win32 build is fine, x64 release buid is fine.

4.2) It could appear with a few of other files as well. But the file could be different depending on my build option (for example, /openmp build may have a different set of files with such errors)

4.3) Now the most mysterious part: if I change the build option a little bit, for example, removing one of /traceback /check:bounds /check:uninit for this file, and manual compile will be fine, but if I restored the /traceback /check:bounds /check:uninit setting as before, manual build would NOT fail anymore. Clean up the whole project and build the whole solution again, the failure will appear again.

The last observation seems to indicate the error is related to some build procedures or environment.

Have anyone seen such a behavior before? Our experience with ifort v13.x has been bad for other crucial issue at the runtime. But teh latest bug build failure is just another thing that becomes a show stopper.

BTW, I have the same issue on Linux build as well where it seems /check:bounds /check:uninit /check:pointer all 3 options will be the trigger to the error. The x64/Windows is a bit different, so far I have not yet found a pattern, except the strange behavior in 4.3.

Any suggestion would be welcome.

[a_zhaogtisoft_com]

Just an update:

I have my PC restored, and then installed VS2010, and then ifort 12.1, and then all the latest components of the parallel studio xe 2013.I am doing this to see whether the trouble is from the many installation (some of the Intel beta over the last 2 years) of the intel parallel studio.

The verdict: the reported problem persists, but with a different set of files!

My PC is using Intel i7-2600 CPU @ 3.4GHz with 16GB.

A similar configured PC (in terms of VS-Parallel Studio) which DO not any the issue is using Intel i7-3770 CPU @ 3.4GHz with 8GB.

Is this a software issue or hardward caused issue?  I do not fully understand the cause of the trouble anymore. But this surely will hit us as the whole company. Typically, if the compiler testing does not pass my test, then no one in our company will initiate the update. So this is very frustrating, because if we could not iron this out, our next software release will be frozen to the ifort 12.1 for the whole release cycle (we have a policy that we do not update the compiler once a major release goes to 'gold', and the compiler version will be frozen throughout the whole life cycle of that release.)

[Steven_L_Intel1]

I wonder if you have a memory problem. That's really the only thing I can think of at this point. It might be interesting to try removing half the memory and see if the symptom changes or goes away. If it's still there, swap with the other half.

[Bernard]

Could be also some hardware problem which can have a catastrophic influence on compiler output.

[a_zhaogtisoft_com]

Steve,

Just did the memory swap in/out test, the crash persists! So it is not memory.

I just asked another developer (who also have the same Sandybridge CPU and Windows 7) to test the project build with latest ifort, he got the same compiler crash:

2>------ Build started: Project: VTRLIB, Configuration: Double Debug x64 ------
2>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.2.190 [Intel(R) 64]...
2>vtr_phas.f
3>------ Build started: Project: MECLIB2, Configuration: Double Debug x64 ------
3>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.2.190 [Intel(R) 64]...
3>mec_sol2.F
2>fit_lookup: Line_seq_number 0000000000000000 was not found
2>fortcom: Fatal: There has been an internal compiler error (C0000005).
2>compilation aborted for C:\s\mihail_solver_v7.4.x\source\vtr\src\vtr_phas.f (code 1)
2>vtr_popv.F
4>------ Skipped Build: Project: lapack, Configuration: Double Debug x64 ------
4>Project not selected to build for this solution configuration
5>------ Skipped Build: Project: xblas, Configuration: Double Debug x64 ------
5>Project not selected to build for this solution configuration
3>fit_lookup: Line_seq_number 0000000000000000 was not found
3>fortcom: Fatal: There has been an internal compiler error (C0000005).
3>compilation aborted for C:\s\mihail_solver_v7.4.x\source\mec\src\mec_sol2.F (code 1)
3>mec_jbrgm.f

I will have to reach out to other developers to test the latest ifort to see how widespread this issue is and if there is any pattern, but from all it seems, this would be a serious road block now.

[Bernard]

>>>3>fortcom: Fatal: There has been an internal compiler error (C0000005).>>>

It looks like access violation error and can be caused indirectly by faulty hardware.

[a_zhaogtisoft_com]

Nope,

I have widen the deployment testing throughout the company's f90 developpers, so far all those who have tried the latest Intel ifort have all reported back the same errors! I am talking about more than 10 F90 developers (using mostly Dell workstation with Sandy Bridge CPU and a few Ivy Bridge CPU)!

An interesting observation from yesterday testing is this: the file sets which had the compiler crash are the same! I am pretty sure they are all having the same source code snapshot. So the previous reproducibility problem (see the 4) in first post) actually seems very deterministic with the same source codes.

This is definitely not a hardware issue, I am going to report to Intel support and request a deeper look. Because this is really hitting us big time or we will have to stay away the whole ifort v13.x release cycle.

[Steven_L_Intel1]

Is this still Premier Support issue 697019?

[a_zhaogtisoft_com]

Steve,

It is still the Premier Support issue 697019 right now.

I think I will have to open a new one considering the importance of issue. Issue 697019 was reported for a similar Linux compiler crash for debug build only, but ultimately I found a work around (i.e., remove one of the options: -check pointer -check bounds -check uninit). The work around is not desirable, but still acceptable. But then we found Windows compiler has the similar crash, all further report was then lumped in.

I think I would report it separately as an Windows ifort issue, instead of letting all those stuck in a Linux issue report.

[Steven_L_Intel1]

Ok, that's a good idea. Please let me know the new issue number.

[Steven_L_Intel1]

We believe that this problem got fixed for compiler version 14.0 in September. Let us know if you're still seeing it.

[Bernard]

Regarding an error description in some cases a misaligned stack (no. of arguments pushed is not equal to no. of arguments poped) could be responsible for access violation exception.