ifx crashes at compilation

[Frank_R_1]

Hi,

First and foremost thank you for fixing sever bugs in ifx compiler which prevent us from switching from ifort to ifx. Finally we can use Intel oneAPI 2025.3.1 ifx on Windows and Linux.

Unfortunately we got random crashes during compilation with error:

catastrophic error: Internal Compiler Error: Ref module: ffe_cl.c

We compile with:

ifx.exe -nologo -fpp -fp=precise -Qimf-arch-consistency:true -Qfma- -MD -bigobj -Qftz -DMSFTZ -nologo -warn:nousage,nounused,declarations,truncated_source,interfaces,general -fpp -names:lowercase -assume:underscore -check:none -O3 -Ob2 -DNDEBUG -Z7 -debug:all -DNDEBUG -module:modules -Qopenmp -c ...

internal error: Please visit 'http://www.intel.com/software/products/support' for assistance.
catastrophic error: Internal Compiler Error: Ref module: ffe_cl.c
compilation aborted for ...

This is the same crash we got randomly with ifort on both platforms. I might have heard that ifx uses the old ifort frontend, which could explain this.

Therefore it would be nice if this also gets fixed, because it sometimes break the build chain.

Best regards

Frank

[Steve_Lionel]

You'll need to provide all the sources necessary to reproduce the problem. Nothing can be done with just a generic error message.

[Frank_R_1]

Hi,

We build parallel with cmake/ninja and get these crashes randomly.

We have a rather huge Fortran code base, and crashes are not file specific! 

But 

catastrophic error: Internal Compiler Error: Ref module: ffe_cl.c

could be a good hint where to look.

Best regards

Frank

[Igor_V_Intel]

Hi Frank,

Please try to reduce the code in that one .f90 file to a small reproducer and share it with us, with the compile options you used.  AM I right that one ifx process never on this file compilation? Unfortunately there is no way to get it fixed based on the source file where ICE happened ( ffe_cl.c). 

[Frank_R_1]

Hi,

As I mentioned, there is no specific .f90 file. It crashes out of the blue during parallel compilation with ninja (about 36 ifx processes running)

This happens ramdomly  and not reproducible!

I have been testing Intel ifort and ifx on Windows and Linux since many years, and the crash was seen with ifort on the first time in parallel studio 20.0.1. Since then we got this problem even after many different oneApi versions till the newest 2025.3.1 ifx.

You should ran it with sanitizer and look if it reveals something in the frontend code of ifx.

A couple of versions of oneApi ago we even got the phenomenon that ifort took all memory during compiling a file and then crashes. This did not happen after 2024.2 again in ifx.

So to conclude I am not able to provide something like a reproducer, I just can encourge you to check the frontend code of ifx.

Best regards

Frank

[Igor_V_Intel]

Do you use IPO(Interprocedural optimization)/LTO(Link Time Optimization)?

"A couple of versions of oneApi ago we even got the phenomenon that ifort took all memory during compiling a file and then crashes. This did not happen after 2024.2 again in IFX." - There were some recent bug fixes in IFX(2025.1, 2025.2) for detected memory leaks and enormous memory consumption.

[Frank_R_1]

No we do not use it. We use the following ifx call (nearly identical on both platforms)

ifx.exe -nologo -fpp -fp=precise -Qimf-arch-consistency:true -Qfma- -MD -bigobj -Qftz -DMSFTZ -nologo -warn:nousage,nounused,declarations,truncated_source,interfaces,general -fpp -names:lowercase -assume:underscore -check:none -O3 -Ob2 -DNDEBUG -Z7 -debug:all -DNDEBUG -module:modules -Qopenmp -c FILENAME

But parallelized calls by ninja build system.

Best regards

Frank

[Frank_R_1]

Hi,

it happens again today:

routine.f90(633): catastrophic error: Internal Compiler Error: Ref module: ffe_cl.c
compilation aborted for routine.f90-pp.f90 (code 1)

Best regards

Frank

[witwald]

Hi @Frank_R_1. I was wondering if you can get CMake to specify that all the Fortran modules created during a build are placed in a unique, per-process directory? The other thing that may be worth considering is to reduce the number of parallel threads, say down from 36 to 8 or 12? I'm not sure what impact that will have on the build time, but it possibly might not be too great, and may lead to greatly improved build robustness. I believe that ninja has a "-j' option that permits one to set the number of parallel jobs, e.g., "-j8". In any case, the link step might be the one that takes a lot of time as well, so reducing  the degree of parallelism in the compilation pipeline may not affect the build time too adversely.

[Frank_R_1]

Hi,

Thank you for your answer, but this is no option. We build a huge code base with lots of C/C++, Fortran, and even Java sources to bundle our product (one CMake/Ninja run). At no time we had problems like this, that parallel compiling/building/linking is the cure. It is only the ifx frontend in parsing Fortran code into indermediate language of llvm. And why parallellism should be a problem? Each Fortran source is a closed entity and gets compiled by one process (or are there some shared dependencies between Fortran compilers during parallel build?). As I pointed out earlier, during different versions of parallel studio and oneAPI we saw other strange behavior like consuming ALL memory and crash of the ifort compiler (on Windows ) as well as the one I posted above on both platforms.

This is a list of errors we collected so far:

Aborting due to.*internal error|error #7001|fortcom: Fatal
Fatal compilation error: Out of memory asking for...
compilation aborted.
internal error: Please visit|catastrophic error: Internal Compiler Error:

Best regards

Frank

[Steve_Lionel]

That only tells us that the error happened during the parsing phase of the compilation. You had not said that the problem was with parallel builds and not simply reproducible.

[Frank_R_1]

Hi,

I don't know if parallel build affects a single ifx compilation (parsing step). ninja spawns multiple ifx processes which each compile one .f90 file. And here we got randomly crashes of the ifx compiler (as we also had with ifort)

Best regards

Frank

[witwald]

@Frank_R_1 I know that you are experiencing "random crashes", and that's very frustrating. However, my (limited) experience is that using '-j36' instead of '-j8' or even '-j4' may counterintuitively not lead to as big reductions in build time as one might expect. Have you tried to run any benchmarks to investigate the build time variation with a range of '-j' settings? With a big, mult-file codebase, that's a bit of a task, but sometimes some testing is required to get a better empirical idea of what's going on "under the hood", so to speak.

Spawning so many parallel compilation tasks can lead to memory usage problems, so, if I may ask, how much physical memory is in the build machine? How many CPU cores does it have? How large is the virtual memory setting?

During a build run, have you kept an eye on the memory usage footprint during the build using Windows Task Manager or similar? What about the distribution of the jobs across the totality of the CPU in your build machine? Have you looked in Task Manager to see the processes that are being spawned by your build job? It would be interesting to see screenshots of data pertaining to all of these types of parameters. It would seem that some of this type of trouble shooting may be in order as the problem is quite complex, and its random nature isn't making finding the root cause easy. Unfortunately, it's not a solution in and of itself, but I hope it helps.

[Frank_R_1]

Hi,

We run builds in our development department on user machines and dedicated build machines under Linux RHEL9 and Windows 10/11 on Intel Xeon and AMD Epyc Machines.

Typical machines have single or dual sockets with RAM from 128GB to 1.5TB and core count is  from 10 (older Xeon single socket) to 128 cores (dual socket Epyc).

The crashes of ifx are on both os platforms and also both cpu platforms!

Memory is no issue (only back when ifort consumes all of 256GB on my machine when compiling 2 files)

To overcome these random crashes and to get to a nightly build on our CI system we instruct CMake/Ninja to rebuild the file again (which then just works)

Best regards

Frank

[Steve_Lionel]

There was a similar issue in the past with the parallel build feature in ifort, where something went wrong with a shared intermediate file (I think). The only thing the devs could "check" is to familiarize themselves with that past problem and see if somehow ifx has repeated it. Just "check the frontend code" is really unfeasible and would push this problem way, way, way down on their stack.

[Shiquan_Su]

Hi, Frank:

Even in a huge code base, the compiler crash is not actually random. The randomness comes from the parallel compilation. And different cores compile different source files in a different speed, and may hit multiple problematic compilation units in different tries. You may switch between parallel compilation and serial compilation to hit the last met compiler crash more reliably.

Let's say you use parallel compilation at the beginning. That will let you compile successfully with most source code files without problems. Then you will hit a compiler crash. Now you switch to the serial compilation, most of the good source code files and object files will not have any change, the serial compilation will always hit that last met crash. Then you can add the verbose output to print the full compilation or linking command of the last crash. When using make, you may "make -j" to parallel compile first, then "VERBOSE=1 make" to serial compile and catch the last compile command in verbose. 

And I see you can hit the error message like this: routine.f90(633): catastrophic error: Internal Compiler Error: Ref module: ffe_cl.c

After you have the verbose command about how your build system compiles the routine.f90, you can play with that command by adding -g -traceback, lowing optimization level, setting different assume behavior ( -assume keyword[, keyword...] ). The error message above even tells you the error is around line 666. So you may further delete other parts in the routine.f90 file if you can still keep the crash happening when you run that verbose compilation command you found above. You may post the remaining problematic part of the routine.f90 file and the verbose compilation command here for further discussion.

Thanks.

Shiquan

[Frank_R_1]

Hi,

We also recently got a stack trace:

#0 0x000055832e7b3f87 f90_abort1 me_globals.c:0 #1 0x000055832e7b40b5 (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x39bc0b5) #2 0x00007f0a6e83ebf0 __restore_rt (/lib64/libc.so.6+0x3ebf0) #3 0x000055832e689207 ss_lm_call_note_sym_per_sym (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x3891207) #4 0x000055832e5215f3 cl_import_use_list ffe_cl.c:0 #5 0x000055832e520be2 cl_import_modules (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x3728be2) #6 0x000055832e607bcd gi_xsorb_ifc_module (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x380fbcd) #7 0x000055832e5b8d20 exCALL (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x37c0d20) #8 0x000055832e5c17f2 exStatement (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x37c97f2) #9 0x000055832e5bca4b exSemExpand (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x37c4a4b) #10 0x000055832e59d76b dxOneSemantic (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x37a576b) #11 0x000055832e59e8f8 dxSemantic (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x37a68f8) #12 0x000055832e7993cd fe_comp_prunit (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x39a13cd) #13 0x000055832e68e34e gem_xx_compile (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x389634e) #14 0x000055832e8753cf main (/data/repo_cache/software/intel/oneAPI/2025.3.1/linux/compiler/2025.3/bin/compiler/xfortcom+0x3a7d3cf) #15 0x00007f0a6e8295d0 __libc_start_call_main (/lib64/libc.so.6+0x295d0) #16 0x00007f0a6e829680 __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x29680) #17 0x000055832e4c623e _start start.S:122

Best regards

Frank

[Shiquan_Su]

Hi, Frank:

Thanks for offering the stack trace. But we can not further investigate just from this stack trace. We need two things. 1. The compilation command right before the error prints out this stack trace. 2. The source code file that the compilation command compiles. Would you please provide those two? Since the error message also says something like:

"""

routine.f90(633): catastrophic error: Internal Compiler Error: Ref module: ffe_cl.c
compilation aborted for routine.f90-pp.f90 (code 1)

"""

The 633 is the line number. You may even remove any sensitive info in the source code file and only keep the part around line 633 with the error. Is it the source code file routine.f90-pp.f90, the file that meets the error and gives you the above stack trace?

Could you please provide the routine.f90-pp.f90 (trimmed if needed) source code file, and the compilation command on routine.f90-pp.f90 which leads to the error and stack trace?