mpiifort issues with F90 file and oneapi

[RRP]

Hi,

I am trying to compile a source code with mpiifort in the oneapi environment as follows:

1) mpiifort -c -real-size 64 -double-size 64 -integer-size 64 *.F90

2) mpiifort -o *  *.o extlibrary.so

I get this error:

ld: /opt/intel/oneapi/compiler/2023.1.0/linux/bin/intel64/../../compiler/lib/intel64_lin/for_main.o: in function `main':
for_main.c:(.text+0x19): undefined reference to `MAIN__'

However, I was not having this issue with the Intel parallel studio 2020 environment.

OS: Rocky 9

 AMD EPYC 7702 64-Core Processor

Thank you in advance.

RRP

[RRP]

Could anybody please help me on this?

Thanks,

RRP

[Barbara_P_Intel]

The error message indicates that you don't have a main program for your application. Is there a main routine in the *.F90 files?

AND this syntax bothers me: mpiifort -o *  *.o extlibrary.so. I don't know why that would have worked with Parallel Studio.

The -o compiler option is followed by a file name. * implies ALL of the files in the directory. Is there more than one file in that directory?

Try:  mpiifort -o exec  *.o extlibrary.so

Here's the reference on -o in the Developer Guide.

[RRP]

- * corresponds to the file name and yes it is in the directory. In this case the name of the executable and the .o file are the same

 - There is no main routine in the *.F90 file

The compiler commands have worked with the Parallel Studio Suite for almost 8 years now, never had any issues.

Now the servers I am on are moving on to newer versions of Linux and hence moving on to the ONEAPI compiler environment. This is a roadblock for me since I need to get this working on that environment too.

Interestingly, the executable created with Parallel Studio Suite and then run in the oneapi environment works but this is not a solution for me moving forward.

I will appreciate any suggestions you might have.

Thanks,

RRP

[RRP]

Barbara,

• MAIN_ is in both object files from old Intel compiler (Parallel Studio 2020) and new oneapi compiler. However, only the oneapi is giving this error
• The fact that it keeps pointing to the 2023.1.0 version in the error here is strange:
  
  ld: /opt/intel/oneapi/compiler/2023.1.0/linux/bin/intel64/../../compiler/lib/intel64_lin/for_main.o: in function `main':
   
• The mpiifort command is only under /opt/intel/oneapi/mpi/2021.9.0/bin
• There are two vars.sh files (that setvars.sh calls), one under 2021.9.0 and one under 2023.1.0. 
• It seems as though this might be causing an issue as the setvars.sh from the Parallel Studio Suite explicitly specified the path to the library and multiple versions of the libraries could co-exist and all had their own mpiifort files
• I couldn't exactly figure out why it chooses the vars.sh in  2023.1.0
• Shouldn't both versions have their own compiler commands

Still stuck, please help.

Thanks,
RRP

[Barbara_P_Intel]

Can you please post a reproducer so I can investigate?

[RRP]

@Barbara_P_Intel 

I have attached the files:

1. source code

2. external library

Compiler commands:

--> source /opt/intel/oneapi/setvars.sh 

--> ifort -c tqex15.F90 -real-size 64 -double-size 64 -integer-size 64 -o tqex15.o

--> ifort -o tqex15  tqex15.o libtq-linux-x86_64-ifort-2023.1.27337-164.so

This is a non-parallelized code but still get the same error as for the parallelized code.

It compiles without any issues with the Parallel Studio Suite 2020.

If it helps, this issue is specific to F90 files. The *.F files do not seem to have any issues.

Thanks,
RRP

[Barbara_P_Intel]

Thank you for the reproducer. Unfortunately, I can't reproduce the failure.

+ ifort --version
ifort (IFORT) 2021.9.0 20230302
Copyright (C) 1985-2023 Intel Corporation.  All rights reserved.

+ ifort -what -c tqex15.F90 -real-size 64 -double-size 64 -integer-size 64 -o tqex15.o
 Intel(R) Fortran 2021.9.0-1497
+ ifort -what -o tqex15 tqex15.o libtq-linux-x86_64-ifort-2023.1.27337-164.so

+ ifx --version
ifx (IFX) 2023.1.0 20230320
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

+ ifx -what -c tqex15.F90 -real-size 64 -double-size 64 -integer-size 64 -o tqex15.o
 Intel(R) Fortran 23.0-1474.2
+ ifx -what -o tqex15 tqex15.o libtq-linux-x86_64-ifort-2023.1.27337-164.so

We use module files in our test environment. I don't need to run setvars separately.

[RRP]

@Barbara_P_Intel 

Thank you for testing.

That is strange. This is what I get with what you did above.

+ ifort --version
ifort (IFORT) 2021.9.0 20230302
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

+ ifort -what -c tqex15.F90 -real-size 64 -double-size 64 -integer-size 64 -o tqex15.o
Intel(R) Fortran 2021.9.0-1497

+ ifort -what -o tqex15  tqex15.o /opt/Thermo-Calc/2023 a/SDK/TQ/libtq-linux-x86_64-ifort-2023.1.27337-164.so
ld: /opt/intel/oneapi/compiler/2023.1.0/linux/bin/intel64/../../compiler/lib/int el64_lin/for_main.o: in function `main':
for_main.c:(.text+0x19): undefined reference to `MAIN__'

+ ifx --version
ifx (IFX) 2023.1.0 20230320
Copyright (C) 1985-2023 Intel Corporation. All rights reserved.

+ ifx -c tqex15.F90 -real-size 64 -double-size 64 -inte ger-size 64 -o tqex15.o
[root@rc-1-rocky9 ONEAPI]# ifx -what -o tqex15  tqex15.o /opt/Thermo-Calc/2023a/ SDK/TQ/libtq-linux-x86_64-ifort-2023.1.27337-164.so
ld: /opt/intel/oneapi/compiler/2023.1.0/linux/compiler/lib/intel64_lin/for_main. o: in function `main':
for_main.c:(.text+0x19): undefined reference to `MAIN__'

I continue to get the same error.

Do you see something odd?

How do I do that with modules?

Thanks,

RRP

[Barbara_P_Intel]

The only thing I can think of is that your environment variable PATH or LD_LIBRARY_PATH is causing this.

Let's not go there with module files. One thing at a time.

[Ron_Green]

@RRP  a few questions:

what gcc version do you have installed on this server?

gcc --version

 Do this version check after you source your oneapi setvars.sh

And you do use bash as your default shell, correct?

And to set env, you do this:
source /opt/intel/oneapi/setvars.sh

Add the -# compiler option to the end of the problem ifx invocation

 ifx -c tqex15.F90 -real-size 64 -double-size 64 -inte ger-size 64 -o tqex15.o -#

and attach the output from the -# output

and can you send the exact version of Rocky 9?

cat /etc/redhat-release

or 

cat /etc/lsb-release

[RRP]

@Ron_Green 

Thank you for looking into this, highly appreciated.

here are the responses to your questions:

gcc --version

gcc (GCC) 11.3.1 20220421 (Red Hat 11.3.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And you do use bash as your default shell, correct?- yes

And to set env, you do this:
source /opt/intel/oneapi/setvars.sh - yes

Add the -# compiler option to the end of the problem ifx invocation

 ifx -c tqex15.F90 -real-size 64 -double-size 64 -integer-size 64 -o tqex15.o -#

and attach the output from the -# output - Attached

cat /etc/redhat-release
Rocky Linux release 9.1 (Blue Onyx)

Best,

RRP

[Barbara_P_Intel]

Thank you.

Now can you do the link step with -#. In the reproducer you gave me that command would be:

ifort -O3 -# -o tqex15 tqex15.o libtq-linux-x86_64-ifort-2023.1.27337-164.so

And post the output from that.

[RRP]

@Barbara_P_Intel 

Output from the command below is attached:

ifort -O3 -# -o tqex15 tqex15.o libtq-linux-x86_64-ifort-2023.1.27337-164.so

Thanks,

RRP

[Ron_Green]

This one is bizarre.  I cannot reproduce what you are seeing.  But I can maybe help you find the issue.

First, the symbol MAIN__ is created by the compiler.  It replaces your PROGRAM statement ( explicit or implicit ) with the symbol MAIN__.  So any source file with a main program gets a MAIN__

take a look my system and your tqext15.F90.  I compile with -c, and use 'nm' on the .o file output and search for MAIN.  Note that in the output below, tqex15.o has a MAIN__ and it's a target "T" meaning that it is defined here in tqext15.o!  This is where the symbol is for MAIN__

ifort -c tqex15.F90 -real-size 64 -double-size 64 -integer-size 64 -o tqex15.o
rwgreen@orcsle153:~/quad/q05847484$ nm tqex15.o | grep MAIN
0000000000000000 T MAIN__

So in your case, use the same commands.  you have to see the same, yes?

Try it with a simple hello world, hello.f90.  Same result, MAIN__ is definded for you in the object file

ifort -c -o hello.o hello.f90
rwgreen@orcsle153:~$ nm hello.o | grep MAIN
0000000000000000 T MAIN__
orcsle153:~$ ifort -o hello hello.o
orcsle153:~$ ./hello
 hello
orcsle153:~$ cat hello.f90
program hello
implicit none

print*, 'hello'

end program hello

Try this with the simple hello.f90 file shown above.  Does this work? It had better work.  If not, I have some ideas.

first, what ld are you using?

which ld

ld --version

alias ld

rwgreen@orcsle153:~/quad/q05847484$ which ld
/usr/bin/ld

ld --version
GNU ld version 2.37-37.fc36
Copyright (C) 2021 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

orcsle153:~/quad/q05847484$ alias ld
-bash: alias: ld: not found

make sure you don't have any aliases and your ld version looks correct for your system (version will be older than mine shown).

Now if the hello world compile test does not show MAIN in the 'nm' output, try these

alias ifort 

alias ifx

This is truly baffling.  You can try to rename tqex.F90 to tgex.f90.  This will remove fpp from the build chain.

which fpp

/opt/intel/oneapi/compiler/2023.1.0/linux/bin/intel64/fpp

alias fpp

 should not have an alias for fpp, make sure it's picking up fpp from the oneapi path.

[RRP]

@Ron_Green @Barbara_P_Intel 

Yes it is bizarre. I have had numerous OS and compiler version updates over the last 8 years and never had any issues with the Intel compiler series.

The MAIN__ is in the object file and always has been but I never received this error with the Parallel Studio Suite compilers.

So I do see the same as you do:

[root@rc-1-rocky9 ONEAPI]# ifort -c tqex15.F90 -real-size 64 -double-size 64 -integer-size 64 -o tqex15.o
[root@rc-1-rocky9 ONEAPI]# nm hello.o | grep MAIN
0000000000000000 T MAIN__

[root@rc-1-rocky9 ONEAPI]# ifort -c -o hello.o hello.f90
[root@rc-1-rocky9 ONEAPI]# nm hello.o | grep MAIN
0000000000000000 T MAIN__

ld --version
GNU ld version 2.35.2-24.el9
Copyright (C) 2020 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

I get "not found" for all the alias commands (ifort, ifx, fpp):

alias ld
bash: alias: ld: not found
[root@rc-1-rocky9 ONEAPI]# alias ifort
bash: alias: ifort: not found
[root@rc-1-rocky9 ONEAPI]# alias ifx
bash: alias: ifx: not found
[root@rc-1-rocky9 ONEAPI]# which fpp
/opt/intel/oneapi/compiler/2023.1.0/linux/bin/intel64/fpp
[root@rc-1-rocky9 ONEAPI]# alias fpp
bash: alias: fpp: not found

I did a fresh install of ONEAPI and now I do not get the MAIN__ error. However, I now run into SEGMENTATION_FAULT errors which also I was never getting before on same machine but compiled with Intel parallel Studio Suite.

Are there any significant changes in ONEAPI for  compiling variable declarations, array sizes, allocable arrays etc?

My backlog of work is piling up and I highly appreciate all the help I am getting here, thank you so much.

Thanks,
RRP

[Ron_Green]

Check stacksize.  

ulimit -a

...

stack size                  (kbytes, -s) 8192

then make sure to open it to unlmited

ulimit -s unlimited

or you can add option

-heap-arrays

try those things.

another thing

compile with -g -traceback

and that'll tell you where the segfault happens.

[RRP]

@Ron_Green 

- Set ulimit -s unlimited

-added option -heap-arrays

I am compiling as follows:

mpiifort -g -traceback -c -heap-arrays filename.F90 -real-size 64 -double-size 64 -integer-size 64 -o filename.o

mpiifort -g -traceback -o filename filename.o libtq-linux-x86_64-ifort-2023.1.27337-164.so

mpiexec -n 1 ./PGM_InitialGrading_EDIT (trying with one core first)

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= RANK 0 PID 11951 RUNNING AT rc-1-rocky9
= KILLED BY SIGNAL: 11 (Segmentation fault)
===================================================================================

I get same error with mpirun.

Thanks,
RRP

[RRP]

@Ron_Green 

Do not seem to get any info on where the segfault is happening.

[Ron_Green]

Old school debug - put a print statement as your first executable statement in your program.  Does it print?

If not

how big are your static arrays?  COMMON, and all arrays declared in your main program?

any idea how much memory you need just to start up the application?

size PGM_InitialGrading_EDIT

what does it say?

have you tried a very small problem size?

Try a MPI hello world.  

PROGRAM hello_world_mpi
include 'mpif.h'

integer process_Rank, size_Of_Cluster, ierror, tag

call MPI_INIT(ierror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size_Of_Cluster, ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD, process_Rank, ierror)

print *, 'Hello World from rank: ', process_Rank, 'of ', size_Of_Cluster, 'total ranks'

call MPI_FINALIZE(ierror)

Does this run? 

[Ron_Green]

Also,

export I_MPI_DEBUG=5

then run