Aborting due to internal error.

[mjdvalens]

Dear reader,

While compiling my Fortran sources with Intel Fortran compiler from OneAPI within Microsoft Visual Studio (for version/installation details see below or the attached file), I regularly encounter a compilation abortion due to an internal error. These errors  seem to happen randomly, in the sense that they may occur on unchanged files that compiled fine in a previous or next compilation attempt. Also they seem to occur in different files from one internal error to another and sometimes there are no errors at all an everything compiles in one go. This also means I have a work-around: just try to compile the exact same code base again. An example of such an error is:

[278/353] Building Fortran object Projects\RFMS\rfms_core\CMakeFiles\rfms_core.dir\src\enter.F90.obj
FAILED: Projects/RFMS/rfms_core/CMakeFiles/rfms_core.dir/src/enter.F90.obj
C:\PROGRA~2\Intel\oneAPI\compiler\20220~1.3\windows\bin\intel64\ifort.exe -ID:\Work\Code\FMS_project\Projects\RFMS\rfms_core\src -ID:\Work\Code\FMS_project\Projects\RFMS\rfms_core\include\rfms_core -ID:\Work\Code\FMS_project\out\build\x64-Debug\Projects\RFMS\rfms_core -ID:\Work\Code\FMS_project\Externals\vincentviewer\Projects\EIMPU\include -ID:\Work\Code\FMS_project\Externals\core\Projects\GULF\include /threads /libs:dll /dbglibs /warn:declarations /warn:unused /warn:interfaces /Qsave /Qinit:zero /Qinit:arrays /fpe:0 /fp:strict /debug:all /Od -c Projects\RFMS\rfms_core\CMakeFiles\rfms_core.dir\src\enter.F90-pp.f90 -o Projects\RFMS\rfms_core\CMakeFiles\rfms_core.dir\src\enter.F90.obj
Intel(R) Fortran Intel(R) 64 Compiler Classic for applications running on Intel(R) 64, Version 2021.5.0 Build 20211109_000000

Copyright (C) 1985-2021 Intel Corporation. All rights reserved.



D:\Work\Code\FMS_project\Projects\RFMS\rfms_core\src\enter.F90(638): internal error : Please visit 'http://www.intel.com/software/products/support' for assistance.

call altent(icdu, alt_, 1, len_, altmin, altmax, TRANSALT_ENTRY)

^

[ Aborting due to internal error. ]

compilation aborted for Projects\RFMS\rfms_core\CMakeFiles\rfms_core.dir\src\enter.F90-pp.f90 (code 1)

Thanks in advance for any suggestions that possibly resolve this issue, kind regards,

Michiel.

P.S. I'm using Microsoft Visual Studio Microsoft Visual Studio Community 2022
Version 17.9.0 with Intel® Fortran Compiler Package ID: w_oneAPI_2022.2.1.14974 The whole about info details are attached (About Microsoft Visual Studio.txt).

[Barbara_P_Intel]

Since you don't have a reliable reproducer, it's hard to offer advice on compiler use. 

Microsoft Visual Studio Community 2022 Version 17.9.0 is so new it was never tested with 2021.5.0.

Have you tried the current compiler release in the HPC Toolkit 2024.0? 

One more thing... ifort is deprecated and will be removed at the end of 2024. The new compiler is ifx. You'll want the latest version of ifx as there have been many bugs fixes since the 2022.1 release.

[JohnNichols]

You can install multiple versions of VS and try them all.  17.9 is very new.   I suggest 2019 as it is well supported. 

I would use the latest ONEAPI as updating recently showed that it improved some of the bugs between VS and Fortran.  

Shifting permanently to IFX is not hard. 

[mjdvalens]

Thank you very much John, I haven't tried the latest ONEAPI yet, I'll have a look!

[mjdvalens]

Thanks again for your advice John. I thought updating looked promising, but unfortunately I still can get the error, even with the 2024.1 version of OneApi (During compilation it shows: Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2024.1.0 Build 20240308). The internal error somehow seems to relate to function calls. I'll keep track of them, hopefully I can get a clue analyzing various errors. 

[mjdvalens]

Thank you very much Barbara, I haven't tried the current compiler release yet, I'll have a look!

[mjdvalens]

Dear Barbara,

Unfortunately we found that the problem still arises with version 2024.1 (part of OneApi 2024.1). Another question that was suggested by a colleague of mine was whether it would be possible to change the compiler verbosity settings such that more details might be shown when the error occurs.

Kind regards,

Michiel. 

[mjdvalens]

Thanks again for the help. I updated the compiler to ifx.exe. At first it looked promising, but unfortunately the error can still occur. I'm using ifx.exe with this specification/version:

Intel(R) Fortran Compiler for applications running on Intel(R) 64, Version 2022.2.1 Build 20221101

[Barbara_P_Intel]

We can best help if you post a small reproducer.

[Jaap0602]

Dear Barbara,

I am a college of Michiel Valens and I want to respond and add some extra info here. 

First of all a small reproduceable is very hard to create and most likely going to mask (or resolve) the problem. My guess is that is has to do with the fact that we are dealing with a very large mixed language project here (433 files, 181 Fortran, rest is C/C++).

The problem came with the introduction of CMake. Before CMake we used VSBuild and we never experienced this 'internal error' before.  A possible reason for this problem may be that CMake does something else like using/starting the compiler in a different way. Since we get the problem at random, it's likely that it has to do with the compiler (parser?) and not with the Fortran code itself.

The internal error only occurs when calling Fortran from Fortran.

The problem occurs often on systems with a very high number of cores. We have 2 build-servers that use Xeon processors:

server1: 1 Xeon Gold 5218R @ 2.1 GHz 64Gb RAM, 40 cores.

server2: 2 Xeon Silver 4218R @ 2.4 GHz 64 Gb RAM,  24 cores (so a total of 48 cores).

On my own computer the problem almost never occurs (Core i7, 12620H @ 2.4GHz, 16 Gb RAM, 16 cores in total)

Could it be that it is some kind of a locking / releasing problem of the compiler?

[Barbara_P_Intel]

How many compiles are happening simultaneously? That is determined by the build system underneath CMake. I ask because you see more failures on the larger configurations. You could be running out of memory, for example.

A workaround is to have the build system cap the number of concurrent jobs at a fraction of the available threads. Maybe start at half. If you continue to have failures, cut in half again. Tune the number of concurrent compiles to get faster builds with no errors.