mpiifort issues with F90 file and oneapi

[RRP]

Hi,

I am trying to compile a source code with mpiifort in the oneapi environment as follows:

1) mpiifort -c -real-size 64 -double-size 64 -integer-size 64 *.F90

2) mpiifort -o *  *.o extlibrary.so

I get this error:

ld: /opt/intel/oneapi/compiler/2023.1.0/linux/bin/intel64/../../compiler/lib/intel64_lin/for_main.o: in function `main':
for_main.c:(.text+0x19): undefined reference to `MAIN__'

However, I was not having this issue with the Intel parallel studio 2020 environment.

OS: Rocky 9

 AMD EPYC 7702 64-Core Processor

Thank you in advance.

RRP

[RRP]

@Ron_Green 

I am programmer for purpose, far from being a pro so yes I did try the old school way :).

No it doesn't run, had already tried that. It is not going beyond the declarations. I didn't have issues like this before, so not sure what it is. 

Had also tried hello_world and it works on all 128 cores with no issues. The output from your file on 1, 2 and 128 cores is attached.

free -h
total used free shared buff/cache available
Mem: 251Gi 3.6Gi 247Gi 17Mi 2.8Gi 248Gi
Swap: 2.0Gi 0B 2.0Gi

size PGM_InitialGrading_EDIT
text data bss dec hex filename
196930 10064 31496 238490 3a39a PGM_InitialGrading_EDIT

setting I_MPI_DEBUG=5 does not provide any information.

Ron, I cannot thank you enough for all this help.

Best,

RRP

[RRP]

@Ron_Green 

Also it is not even running on a single core which should have access to the whole memory.

So if shared memory is the limiting factor here while running on multiple cores, it should run on a single core easily. The executable is not memory heavy and usually uses a max of 8-10 GBs even with multiple cores.

Thanks,

RRP

[RRP]

@Ron_Green 

Another update:

The executable compiled with the Parallel Studio Suite 2020 runs fine with no issues.

However, the compiled version with oneapi continues to fail.

Sorry for the multiple updates but have been trying numerous ways to get this sorted and to provide you as much information as possible.

Thanks,

RRP

[Ron_Green]

the data is good.

MPI works for hello world, and to scale.

you never get to the print statement.  This means the code is dying in startup.  Two possibilities

  - your .so library is causing the crash

  - you have too much static data, if not in your main program then in the .so startup.  Does your code have COMMON block, BLOCK DATA, and or array declarations like

       program onetwothree

       ....

  INTEGER, PARAMETER :: N=10000      

  REAL*8  BIGSTATIC( N, N, N )

   or is your data ALLOCATABLE ? 

   Static data is limited on may OSes to 2GB total, even with 64bit OS and address.  Static data size is fixed.  Allocatable is only limited to RAM size + SWAP size.  But you don't get to the print so you probably are not calling ALLOCATE, correct? 

At startup, that .so file is loaded.  IF that crashes during the load you will see the behavior you have.  Same for oversized static data.  

One oddity I noticed on the .so you sent that I didn't mention previously is this: the 'strings' output shows that the .so you included with the example may have objects that were compiled with an old v16 compiler?  'strings' gives a lot of output, here is what I did:

strings libtq-linux-x86_64-ifort-2023.1.27337-164.so |& tee strings.txt

then just brought it up into an editor and searched for 'IFORT' and 'Fortran' 

ifort (IFORT) 16.0.4 20160811

...
CONSIS
NOT POSSIBLE TO RUN THIS IN GES 6
03-02-2023 09:51:44
Linux
27337
ifort (IFORT) 16.0.4 20160811
THCD
...

Is this .so comprised of LAPACK and other libraries that were built with the v16.0 compiler?  We do not officially support compatibility of binaries across major versions.  However, in more modern iforts, we've kept compatibility.  But not all the way back to v16.   V19, sure.  V16 well ... no guarantees.  Plus, these old binaries may be making system calls into the linux OS that have changed or been removed since 2016.  
Any chance you can recompile and rebuild the .so with objects from the 2023 compiler, or at least v19 or above?

[Ron_Green]

OH one other thought

which mpiexec

which mpirun

where are you picking up mpiexec?  

You use Intel MPI, correct?  Intel MPI recommends MPIRUN these days.  Also, some servers have OpenMPI or MPICH installed in default locations.  Let's make sure you are getting mpiexec/mpirun from the  /opt/intel/oneapi/mpi/2021.9.0/bin directory. 

Since hello world ran, these mpirun/exec command are probably correct.

[Ron_Green]

and any chance you can move the .exe file and the .so's from this server to a server with an older OS? 

Try removing the Rocky variable.  Rocky "should be" OK, but who knows.  It only came out officially May 16th 2023! 

The compiler "should be" ok, although we do not officially support Rocky. And for sure we have not qualified our compiler for 9.2, it's too new.  I'll check with my MPI friends on any known RHEL 9.2 and Rocky 9.2 issues.

[RRP]

Any large variables are allocatable which shouldn't cause an issue, since it doesn't get to the print statement before "allocate", correct?

It is possible that the .so file was built with an older version (most likely Fortran composer 16). I have no way of changing that since this is a file from the software vendor.  Any workaround there?

[root@rc-1-rocky9 cloud]# which mpirun
/opt/intel/oneapi/mpi/2021.9.0/bin/mpirun
[root@rc-1-rocky9 cloud]# which mpiexec
/opt/intel/oneapi/mpi/2021.9.0/bin/mpiexec

This seems to be correct.

[Ron_Green]

Actually I had a thought on the .so file

can you write a new, simple, Fortran program that calls just 1 function in that .so library?  And compile/link to the library.  

Then run it without MPI

Then run it with

mpirun -np 1 ./test

this will tell us if Rocky does not like the .so file.  I will research if there are other ways to detect .so file load failures.

[Ron_Green]

and a simple test for LD_LIBRARY_PATH

mpiexec -n 1 echo $LD_LIBRARY_PATH

/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/ipp/2021.8.0/lib/intel64:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/mkl/2023.1.0/lib/intel64:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/debugger/2023.1.0/gdb/intel64/lib:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/debugger/2023.1.0/libipt/intel64/lib:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/debugger/2023.1.0/dep/lib:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/tbb/2021.9.0/lib/intel64/gcc4.8:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/compiler/2023.1.0/linux/lib:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/compiler/2023.1.0/linux/lib/x64:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/compiler/2023.1.0/linux/compiler/lib/intel64_lin:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/ccl/2021.9.0/lib/cpu_gpu_dpcpp:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/mpi/2021.9.0/libfabric/lib:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/mpi/2021.9.0/lib/release:/nfs/pdx/disks/cts2/tools/oneapi/2023.1.0/mpi/2021.9.0/lib:/cts/tools/tcl/lib:/cts/tools/tcl/lib

This is the load library path in the environment run by MPI. This path needs to have the path to the .so library.  Do you see the path to your .so library in the output of 'echo $LD_LIBRARY_PATH'?   If the path to the .so is missing, this could be a simple issue to fix.

[RRP]

- Yes the path to .so library is included in LD_LIBRARY_PATH

[RRP]

Yes, I can do that but the examples I had shared earlier did run without issues although it was a serial run. This one had much less variables and is not as elaborate as the one I am having issues with.

[Ron_Green]

Did you run that simple serial one you shared here under mpiexec or mpirun?  Even serial code can be launched by mpirun or mpiexec

mpirun -np 1 <simple case executable>

does it run under the mpi launcher?

[Steve_Lionel]

Regarding static data size - only Windows limits this to 2GB. On Linux (and probably Mac), the default limit is 2GB but if you use "-mcmodel medium" you can get much larger static data. (-mcmodel large gets you large data and code.) That said, I recommend use of allocatable arrays for large data.

[tanzania567]

Review and verify the compiler options you are using, making sure they are supported by the oneAPI Intel Fortran compiler and set correctly.

[alicja01]

thanks for this information