ifort (IFORT) 2021.10.0 20230609 vs ifort (IFORT) 2021.4.0 20210910

[MircoValentini]

I am trying to build a library with ifort 2021.10.0 on a Linux HPC cluster and have encountered several issues:

1) Previous Success with ifort 2021.4.0:

•  The library was building and running smoothly with ifort 2021.4.0.

2) Issues with ifort 2021.10.0:

• The build process hangs for a while.
• After some time, an internal compiler error appears.

3) *Potential .mod File Generation Issue:

• The .o files are generated successfully (still need to check if the symbols are there).
• However, the corresponding .mod files are not being generated.

4) Complex Code Structure:

• The files implement many specializations of an abstract class and a factory used to allocate different extended types.
• There is a very long list of USE :: <MODNAME>, ONLY: <USED_SYMBOL> statements.
• Commenting out some of these USE statements (and the related code) changes the error message and its position in the code. 
• Commenting out only the related code and not the USE statements doesn't change the error message

5) Difficulty in Creating a Small Reproducer:

• The issue seems related to the long list of USE statements, making it challenging to create a small, isolated reproducer. Investigation is ongoing.

6) Performance Degradation:

• When building the extended types (one extended type per module and one module per file), the new compiler (ifort 2021.10.0) is considerably slower than the previous version (ifort 2021.4.0).

7) Build flags:

•  The problem happens both in release and debug mode (with and without checks)

ifx behaviour:

• The problems seems to happen (in a similar way) also with ifx 20230622

Any suggestion?

[andrew_4619]

Before spending too much time why not verify this issue with 2021.12.0 the problem may have gone away. The description of the problem  does not help that much without some sample code. You say "changes the error message and its position in the code" maybe showing the error messages might help?  It could be the simple fact that 'a lot'  of use....only  statements is exposing a compiler limit  issue. I had a similar (but different) problem recently and wrote a simple program to autogenerate a lot of subroutines and interfaces  to them  as a reproducer. 

How long is a long list BTW?

[Ron_Green]

We do not have enough details to help here.  As @andrew_4619 said it would be good to try the latest compiler, 2024.1. 

Also, you can try adding this compiler option

-qoverride-limits

Is this code publically available?  And I'm curious, does it use NetCDF or HDF5?  Is the internal compiler error (ICE) in your code or a 3rd party library build like NetCDF.  If the code is publically available we can investigate.

As for testing the latest compiler - I understand that cluster admins are slow to update their SW stacks.  But you can install your own local copy of Intel Fortran for a test.

First, since your home space is probably under quota, and you probably have a project or scratch space, do this to avoid blowing over your home dir quota during the local install.  Lets say your project space is an nfs space named '/projects/mine

cd /projects/mine

mkdir intel

cd

ln -s /projects/mine/intel  intel

Now ~/intel is a link to your project space with a lot of free space to use ( say 4GB or more )

Next, download just the Fortran compiler using this guide for directions.   Run the *.sh file to start the installer.

During the install, select CUSTOMIZE

select the Fortran compiler

Change the installation directory to your project space, maybe something line /projects/mine/intel-compilers

The installation keeps a database of past installations.  That will be saved in ~/intel directory.  So when it's time to remove all this, simply

rm -Rf ~/intel/*

Then remove /projects/mine/intel-compilers directory

[MircoValentini]

UPDATE: after a code reorganization, I am able to build and run. However, there issue still persist in the original code.

@andrew_4619: 

1) ... why not verify this issue with 2021.12.0 ... -> I tested this on my workstation, but the compiler hangs indefinitely.

2) The description of the problem  does not help that much without some sample code -> I am unable to provide a (reasonably) small reproducer. Below you can find a link to the problematic code.

3) ... maybe showing the error messages might help? -> The behavior is inconsistent; sometimes the compiler hangs without messages, other times it results in an Internal Compiler Error (ICE), or prints unreasonable errors in random places within the module.

4) How long is a long list BTW? -> Almost 70 USE lines.

@Ron_Green 

1) ... to try the latest compiler, 2024.1 -> tested on my personal machine, the compiler hangs indefinitely.

2) -qoverride-limits -> interesting, I wasn't aware of this option, thanks for pointing it out.

3) Is this code publically available? -> The entire code isn't public, but here is the file causing issues: https://github.com/ecmwf/multio/blob/2.1.5/src/ecom/encoders/grib_encoder_manager_mod.F90

The problem seems related to  "USE :: ENCODERS_UTILS_MOD, ONLY: UTILS_INIT"

4) And I'm curious, does it use NetCDF or HDF5? -> The ICE is occurring within my code, not involving NetCDF or HDF5.

5) Thanks for the suggestions. I have the latest compiler version installed on my personal machine.

Thanks for the support.

[Ron_Green]

I am curious if ifort 2021.12.0 testing made any difference.

2021.13.0 will appear with the next update release, oneAPI 2024.2.  This is due out late June or early July.  

Does the application use OpenMP?  Have you tried turning it off.  Leave -qopenmp off of the compilation options OR use -qopenmp-stubs to stub out all the OMP API calls and directives.  Wondering if OpenMP is a trigger

Did qoverride-limits help?  usually this just delays the eventual ICE but sometimes it's worth a try

Parameterized Derived Types in your code?  We have a few known bugs in this area.  Some will be fixed in 2024.2, and the rest to be addressed in the 2025.0 package BUT remember that ifort will not by default be in the oneAPI 2025.0 packages. 

[MircoValentini]

• I am curious if ifort 2021.12.0 testing made any difference.
  Instead of an ICE, the compiler hangs forever.

• 2021.13.0 will appear...
  I will try it as soon as it becomes available.

• Does the application use OpenMP?
  No, it does not use OpenMP or MPI. It is just a library that is called inside an MPI software.

• Did qoverride-limits help?
  I haven't tried it yet. After a code reorganization (mainly related to the USE hierarchy), I am now able to build with both ifort and ifx. I plan to give it a try in my spare time out of curiosity. However, I don't think it would be safe to use such a flag in production.

• Parameterized Derived Types in your code?
  No, we have a wide CI on GitHub, and I spent the last weeks trying to make all compilers happy. This process required removing almost all "advanced" Fortran 2003 and 2008 features. The only "advanced" feature I am still using and rely on is basic polymorphism.