mixed mpi/coarray programs - is -coarray-config-file required?

[AShte]

I've a mixed mpi/coarray program that I'm trying to port to intel. I build it with mpiifort: mpiifort --version ifort (IFORT) 15.0.2 20150121 but I'm very confused about  -coarray-config-file option. I can build the executable with or without this option, but so far I haven't managed to run it, with or without this option. My latest attempt is to specify -coarray-config-file at link time, and submit the job to PBS queue with: #PBS -l walltime=01:00:00,nodes=4:ppn=16 mpdboot --file=$PBS_NODEFILE -n 1 mpdtrace -l prog data mpdallexit where "prog" is the executable, and "data" is the input data for it. The coarray config file has: -envall -n 64 prog data The results are weird - seemingly both coarray and MPI parts think there are only 16 cores available (a single node). num_images returns 16, and only 16 MPI processes are created. This makes me think that somehow the fact that I build with -coarray=distributed is not being acknowledged. What else can I try? Is MPI/coarray programming still unsupported in ifort 15? Thanks Anton

[AShte]

I think I got it, but would be great if somebody from Intel confirms my observations/conclusions. It seems MPI init/finalize cannot appear in a coarray program. This program works as intended: program test implicit none include 'mpif.h' integer :: rank, PEs, errstat, img, nimgs !call MPI_INIT( errstat ) call MPI_COMM_SIZE( MPI_COMM_WORLD, PEs, errstat ) call MPI_COMM_RANK( MPI_COMM_WORLD, rank, errstat ) write (*,"(2(a6,i3))") "PEs:", PEs, "rank:", rank img = this_image() nimgs = num_images() write (*,"(2(a6,i3))") "nimgs: ", nimgs, "img:", img !call MPI_FINALIZE( errstat ) end program test Compiled with mpiifort -coarray=distributed -coarray-config-file=test.conf test.f90 where $ cat test.conf -envall -n 20 a.out on runtime gives: $ ./a.out PEs: 20 rank: 1 nimgs: 20 img: 2 PEs: 20 rank: 3 nimgs: 20 img: 4 PEs: 20 rank: 4 nimgs: 20 img: 5 PEs: 20 rank: 6 nimgs: 20 img: 7 PEs: 20 rank: 2 nimgs: 20 img: 3 PEs: 20 rank: 12 nimgs: 20 img: 13 PEs: 20 rank: 0 nimgs: 20 img: 1 PEs: 20 rank: 18 nimgs: 20 img: 19 PEs: 20 rank: 7 nimgs: 20 img: 8 PEs: 20 rank: 16 nimgs: 20 img: 17 PEs: 20 rank: 17 nimgs: 20 img: 18 PEs: 20 rank: 5 nimgs: 20 img: 6 PEs: 20 rank: 13 nimgs: 20 img: 14 PEs: 20 rank: 14 nimgs: 20 img: 15 PEs: 20 rank: 11 nimgs: 20 img: 12 PEs: 20 rank: 15 nimgs: 20 img: 16 PEs: 20 rank: 9 nimgs: 20 img: 10 PEs: 20 rank: 10 nimgs: 20 img: 11 PEs: 20 rank: 19 nimgs: 20 img: 20 PEs: 20 rank: 8 nimgs: 20 img: 9 which is as expected. If I uncomment MPI_init and MPI_finalize, I get lots of Attempting to use an MPI routine after finalizing MPI So this seems clear. If I remove -coarray-config-file, then the execution is limited to a single node (16 cores in my case), I guess because there is nothing to tell the program how many cores to use, since mpirun is not involved. Thanks Anton

[Steven_L_Intel1]

By default, we try to do our own MPI init. Try compiling the main program with "-coarray=single" and see if that helps. This is generally what you need to do if mixing with your own MPI.

[AShte]

Did I get you right? $ cat test.f90 program test implicit none include 'mpif.h' integer :: rank, PEs, errstat, img, nimgs call MPI_INIT( errstat ) call MPI_COMM_SIZE( MPI_COMM_WORLD, PEs, errstat ) call MPI_COMM_RANK( MPI_COMM_WORLD, rank, errstat ) write (*,"(2(a6,i3))") "PEs:", PEs, "rank:", rank img = this_image() nimgs = num_images() write (*,"(2(a6,i3))") "nimgs: ", nimgs, "img:", img call MPI_FINALIZE( errstat ) end program test $ mpiifort -coarray=single test.f90 $ mpirun -n 4 ./a.out PEs: 4 rank: 1 nimgs: 4 img: 2 PEs: 4 rank: 0 nimgs: 4 img: 1 PEs: 4 rank: 2 nimgs: 4 img: 3 PEs: 4 rank: 3 nimgs: 4 img: 4 Attempting to use an MPI routine after finalizing MPI Attempting to use an MPI routine after finalizing MPI Attempting to use an MPI routine after finalizing MPI Attempting to use an MPI routine after finalizing MPI Something is still wrong.... Anton

[Steven_L_Intel1]

You don't need the MPI_FINALIZE call, I think. I will ask our expert in such things, Lorri, to chime in.

[Lorri_M_Intel]

Aw jeez ... so much pressure.

Steve is right, it is safe to remove the MPI_Finalize call. 

Our run-down code also calls MPI_Finalize, so your program will be appropriately "finalized".

For those who are curious:  the MPI calls that are being made in our run-down code (the ones that provoked the runtime errors in the example program) are to synchronize memory before image termination.

           I hope this helps --

                      --Lorri

[AShte]

My experiments suggest otherwise. If I try to build an MPI program with -coarray=single, and then run the executable with mpirun, I get strange behaviour - as if some processes stall after a while. The test code above does indeed work, but a larger project behaves weirdly. However, if I think of a hybrid MPI/coarray program as a coarray program with some MPI commands, I can get to run the executable ok. In this case I do: 1) remove bothMPI_Init and MPI_Finalize 2) compile and link with -coarray=distributed -coarray-config-file=config.file and 3) run with no mpirun, but simply using the executable name, just as any other coarray program with distributed memory. This seems to work a lot better. Anton