x64 debug build failure with "fit_lookup: Line_seq_number 0000000000000000 was not found"

[a_zhaogtisoft_com]

I have been struggling with this strange error when building debug version of the our solver using Fortran Compiler XE 13.1.1.171. This issue has already been reported to Intel support,but I have not heard anything from them, and I hope someone here can give me a hint - the issue literately block any of Intel ifort v13.x deployment, if not addressed, we will be stuck to ifort v12.1.

1) Build environment: Fortran Compiler XE 13.1.1.171 hosted on VS2010 (part of the Intel Parallel Studio) on Windows

2) Target: x64 only (with or without -openmp), debug build only

3) Build error:

Compiling with Intel(R) Visual Fortran Compiler XE 13.1.1.171 [Intel(R) 64]... Creating temporary file "RSP1.rsp" with contents [ /nologo /debug:full /Od /fpp /I".." /I"../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\lib\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\chem\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /assume:nosource_include /DDEBUG /reentrancy:threaded /extend_source:132 /real_size:64 /align:rec16byte /align:qcommons /align:sequence /fpe:0 /fpconstant /names:lowercase /iface:cref /assume:underscore /module:"../modules/x64/Double Debug/" /object:"x64\Double Debug/" /traceback /check:bounds /check:uninit /libs:static /threads /dbglibs /c /extfor:f /Qvc10 /Qlocation,link,"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\\bin\amd64" "C:\build\V740\source\cnt\Src\cnt_summ.f" ] Creating command line "ifort @"C:\build\V740\source\cnt\x64\Double Debug\RSP1.rsp"" fit_lookup: Line_seq_number 0000000000000000 was not found fortcom: Fatal: There has been an internal compiler error (C0000005). compilation aborted for C:\build\V740\source\cnt\Src\cnt_summ.f (code 1) Creating temporary file "RSP1.rsp" with contents [ /nologo /debug:full /Od /fpp /I".." /I"../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\lib\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /I"C:\build\V740\source\chem\../modules/x64/Double Debug/" /I"C:\build\V740\source\modules\../modules/x64/Double Debug/" /assume:nosource_include /DDEBUG /reentrancy:threaded /extend_source:132 /real_size:64 /align:rec16byte /align:qcommons /align:sequence /fpe:0 /fpconstant /names:lowercase /iface:cref /assume:underscore /module:"../modules/x64/Double Debug/" /object:"x64\Double Debug/" /traceback /check:bounds /check:uninit /libs:static /threads /dbglibs /c /extfor:f /Qvc10 /Qlocation,link,"C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\\bin\amd64" "C:\build\V740\source\cnt\Src\cnt_dlay.f" ] Creating command line "ifort @"C:\build\V740\source\cnt\x64\Double Debug\RSP1.rsp"" CNTLIB - 1 error(s), 0 warning(s)

4) Reproducibility:

4.1) For above file, it always fails with the same error output, but only limited to x64 debug build. Win32 build is fine, x64 release buid is fine.

4.2) It could appear with a few of other files as well. But the file could be different depending on my build option (for example, /openmp build may have a different set of files with such errors)

4.3) Now the most mysterious part: if I change the build option a little bit, for example, removing one of /traceback /check:bounds /check:uninit for this file, and manual compile will be fine, but if I restored the /traceback /check:bounds /check:uninit setting as before, manual build would NOT fail anymore. Clean up the whole project and build the whole solution again, the failure will appear again.

The last observation seems to indicate the error is related to some build procedures or environment.

Have anyone seen such a behavior before? Our experience with ifort v13.x has been bad for other crucial issue at the runtime. But teh latest bug build failure is just another thing that becomes a show stopper.

BTW, I have the same issue on Linux build as well where it seems /check:bounds /check:uninit /check:pointer all 3 options will be the trigger to the error. The x64/Windows is a bit different, so far I have not yet found a pattern, except the strange behavior in 4.3.

Any suggestion would be welcome.

[a_zhaogtisoft_com]

It looks like the copy of text from build log is scrambled. So here is the error output:

3>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.1.171 [Intel(R) 64]...
3>cnt_summ.f
3>fit_lookup: Line_seq_number 0000000000000000 was not found
4>------ Build started: Project: MECLIB2, Configuration: Double Debug x64 ------
3>fortcom: Fatal: There has been an internal compiler error (C0000005).
4>Compiling with Intel(R) Visual Fortran Compiler XE 13.1.1.171 [Intel(R) 64]...
3>compilation aborted for C:\build\V740\source\cnt\Src\cnt_summ.f (code 1)

[Steven_L_Intel1]

Please provide us with a ZIP of the source in question.  You can attach it here, use "Send author a message" to provide it to me privately, or use Intel Premier Support. Please be sure to include any include files or module sources it needs.

You may want to try turning off "Check Routine Interfaces" under Diagnostics to see if that makes a difference. The reproducibility issues you note make me suspect it is relevant.

[a_zhaogtisoft_com]

Hi, Steve,

There is already a test case (MSVS project) submit to Intel Premier Support (IssueID : 697019). It was originally created for similar Linux build failure, but the reproducibility issue there prevented me to create a test case for isolated file.

I just check the "Check Routine Interfaces", it is off all the time.

Let me know what else I can help.

[a_zhaogtisoft_com]

In addition to the "fit_lookup: Line_seq_number 0000000000000000 was not found" error output, there is another type of error also showing up from time to time:

4>This application has requested the Runtime to terminate it in an unusual way.
4>Please contact the application's support team for more information.
4>GEM_LO_GET_LOCATOR_INFO: zero locator value
4>C:\Users\ALLEN~1.GTI\AppData\Local\Temp\72763.i: catastrophic error: **Internal compiler error: abort signal raised** Please report this error along with the circumstances in which it occurred in a Software Problem Report.  Note: File and line given may not be explicit cause of this error.
4>compilation aborted for C:\build\V740\source\eng\src\eng_fmac.f (code 1)

"GEM_LO_GET_LOCATOR_INFO: zero locator value" feels more un-predictable if not the same.

But the behavior observed in 4.3) is applicable here. If I remove /traceback, the compiler crash is gone, restore the setting, the compiler crash is not coming back. Totally bizarre.

[jimdempseyatthecove]

Remove your option of choice such that you can build.

Perform Clean

Perform Build All

Do you receive an error report indicating one of the .mod files cannot be found?
(Do not re-issue Build until error goes away and be satisfied that the Build is OK).

Circular dependencies in .mod files sometimes gives wierd results. Fix the circular dependency. Then restore options that gave you problems. Then Clean, and Build All and see if problem comes back.

Jim Dempsey

[a_zhaogtisoft_com]

Jim,

I doubt we have any .mod circular dependencies: we have several other configurations (for example: Debug | Win32), they just build fine, never reporting any errors with your suggested steps.

Steve,

What really bugs me at this stage is the observation that this Debug|x64 issue only happens on my PC. With the same source codes, I can not even reproduce it on a different machine that also have ifort 12.1 and 13.1 installed. Of course, my PC has a lot of demo and extra compiler installed (I have 12.1, 13.0, 13.1 and 14.0 all installed).

I even did a diskcheck on my pc to see if that helps. Well, it didn't. Some registry messed up or stuff that we do not know? Not sure.

[Steven_L_Intel1]

In cases such as this I usually suggest rebooting into "Safe Mode" and see if you can reproduce the problem. If not, then that suggests some background process is interfering with the compilation. You can then use msconfig and the Startup tab to selectively disable startup options and see if you can identify the culprit.

[jimdempseyatthecove]

What Steve is suggestion (though not explicitly stated) is the situation where the share mode of a compiler output/input/temp file is such that when an Anti-Virus opens the file for scanning, that this interferes with the compilation process. This had been a reported problem (which may not have been fixed in the version of the compiler/linker you are using). This may not only be Anti-Virus. Once you identify the culprit, Intel software support may be able to engineer a fix.

Jim Dempsey

[Steven_L_Intel1]

If it is an external program interfering, there is likely nothing we can do about it. My experience is that problems that can be reproduced on one computer only are due to some influence outside the compiler product. I have even seen a case where a coding error in a printer driver caused problems!

[a_zhaogtisoft_com]

Just tested the idea with safe-mode (without networking), the problem persists, both errors "fit_lookup: Line_seq_number 0000000000000000 was not found" and "GEM_LO_GET_LOCATOR_INFO: zero locator value" are encountered, and what is interesting and puzzling is that file set that having above errors are different from those observation when building in normal mode.

[a_zhaogtisoft_com]

Just found out ifort 2013.4.190 was released several days ago.

So I went through an excise that I uninstalled all Intel components (Parallel Studio, all compiler update) from my PC, and then re-installed only following:

1) ifort 2011.4.196 (this is by far the only version that works!)

2) All latest Parallel Studio components (ifort/icc/icpc 2013.4.190, VTune 2013 update 7, Inspector Update6, and Advisor Update 3).

Well, the bad news is that on this system only 2 version of ifort exist, the same old errors still persist.

I am really not sure what is causing this now. I think tomorrow, I will have my PC reformated to factory setting, and start from scratch.

[jimdempseyatthecove]

Not that there should be anything wrong with this, your .rsp file contains:

C:\build\V740\source\modules\../modules/x64/Double Debug/

While it should not matter if your path goes up and down ("modules\../modules"), do your working configurations have that peculiarity?

The above syntax may require the modules folders to contain the symbolic link (..). In some cases this is not always true.

Jim Dempsey

[a_zhaogtisoft_com]

Jim,

I do not recall we ever use .. symbolic link. The build project has been working for more than 8 years for Win32, and 1 year for x64 build; it is just my observation, the ifort is taking either / or \ for path separator, never giving me trouble.

For the output I have shown, the project is having "..;..\modules\$(PlatformName)\$(ConfigurationName)\" or "..;../modules/$(PlatformName)/$(ConfigurationName)/" under Fortran->General->Additional Include Directories and Fortran->Output Files->Module Path.

I just changed the / to \ for this particular project, but somehow the command line display still has C:\build\V740\source\modules\../modules/x64/Double Debug/, not sure how this happened.

The bottom line, after I changed / to \ under Fortran->General->Additional Include Directories and Fortran->Output Files->Module Path, nothing changed for the build, same error for cnt_summ.f.

[Bernard]

It you suspect an AV interefering with compilation process I think that booting into safe mode will not help.Filter drivers belong to AV file scanning could be marked as a boot drivers.

[jimdempseyatthecove]

Most AV programs have an option to temporarily disable scanning (or manually turn off/on). Barring that you may be able to (temporarily) specify your output folders (temp, obj, mod, ...) as not to be scanned. If AV is not the issue then set everything back.

Jim Dempsey

[a_zhaogtisoft_com]

I think AV is not an issue: we have the same failure for roughly the same file sets when running in windows safe mode (without anything running)

[a_zhaogtisoft_com]

just uninstall the Symantec endpoint protection entiredly from my PC, I still run into the same issues. So it is definitely not an AV issue.

BTW, did I forget to mention that on Linux (RHEL5.8), we are seeing the similar type of errors with ifort v13.1? We definitely have no AV there at all. And we found out on Linux, when debug build is using "-check pointer -check bounds -check uninit" together we will be hit by the error

fit_lookup: Line_seq_number (nil) was not found

I have always seen these troubles are related.

[Lorri_M_Intel]

I don't think it's related to an external program; I have seen this error in previous releases when Something Bad happens during the time after the compiler starts up and before all its internal initializations have happened.  

For this particular source file, I got it to compile by removing the requirement for /fpp (commented out the #if / #endif).   You'll have to create the correct workaround of course; maybe use !dir$ if / !dir$ endif pairs instead?

I'm still working at getting it to reproduce in an environment that I can debug the compiler.

                    --Lorri

[a_zhaogtisoft_com]

taking off /fpp for this cnt_summ.f does make the compile going through.

But the effect of /fpp behaves like what I have described in 4.3) in the first post. Restoring /fpp, the file still build fine. One of those things that make me pull my hair....

[Bernard]

jimdempseyatthecove wrote:

Most AV programs have an option to temporarily disable scanning (or manually turn off/on). Barring that you may be able to (temporarily) specify your output folders (temp, obj, mod, ...) as not to be scanned. If AV is not the issue then set everything back.

Jim Dempsey

 That is true did not think about this option.