mpi help

[roddur]

dear friends,
i am trying to run a code using mpi. i have done simple prigrams before but thistime iam getting several confusion. plz help. in my main program, i have initialized mpi as

ierr = ppinit(idproc, nproc)
if (ierr.ne.0) stop ! stop right there if there's a problem
print *, "I'm processor #", idproc, " in a ", nproc, "-processo r cluster."

!$omp parallel
call DoParallel()
!$omp end parallel
call ppexit()
where ppinit and ppexit is a startup and ending the mpi enverionment.
module use_mpi
include 'mpif.h'
CONTAINS
!***********************************************!
! Subroutine to init mpi !
!***********************************************!
function ppinit(idproc, nproc)
implicit none
! this subroutine initializes parallel processing
integer :: idproc ! = processor id
integer :: nproc ! = number of real or virtual processors obtained
integer :: ierr, ppinit

nproc=0

! initialize the MPI execution environment

call MPI_INIT(ierr)
if (ierr.eq.0) then

! determine the rank of the calling process in the communicator
call MPI_COMM_RANK(MPI_COMM_WORLD, idproc, ierr)

! determine the size of the group associated with a communicator
call MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr)

end if

ppinit = ierr

end function

!***********************************************!
! Subroutine to terminate mpi
!***********************************************!
subroutine ppexit()
implicit none
integer :: ierr
call MPI_Finalize(ierr)
end subroutine

end module

i have not yet implimented any other mpi directive and compiled the code usiing make as:
mpif90 -c -O3 -openmp main.f90
and running as
$ mpiexec -np 4 ./irun
its then prompting some line which is completly outside the mpi region(write statement multiple time).
is there somthing i am missing?

[rreis]

Quoting - roddur

i have not yet implimented any other mpi directive and compiled the code usiing make as:
mpif90 -c -O3 -openmp main.f90
and running as
$ mpiexec -np 4 ./irun
its then prompting some line which is completly outside the mpi region(write statement multiple time).
is there somthing i am missing?

Just one question, if you don't use OPENMP (compile without the -openmp flag) does it run ok? And does it run OK if you comment the MPI subroutines and just compile with ifort (or other fortran compiler)?

[roddur]

Quoting - rreis

Just one question, if you don't use OPENMP (compile without the -openmp flag) does it run ok? And does it run OK if you comment the MPI subroutines and just compile with ifort (or other fortran compiler)?

If you are asking if it ran using openmp alone(without mpi), and in serialmode, the ans. is yes.

[rreis]

Quoting - roddur

Quoting - rreis

Just one question, if you don't use OPENMP (compile without the -openmp flag) does it run ok? And does it run OK if you comment the MPI subroutines and just compile with ifort (or other fortran compiler)?

If you are asking if it ran using openmp alone(without mpi), and in serialmode, the ans. is yes.

yes, and also:

- run with NO openMP, just MPI and serial (-np 1)
- run with NO openMP, just MPI and parallel

[TimP]

When you enable OpenMP along with MPI, you must change from MPI_INIT to MPI_INIT_THREAD, checking whether the type of threading support you require is enabled.
You would require Intel MPI 3.2 to get affiinity support, engaged by setting I_MPI_PIN_DOMAIN=omp

[roddur]

Quoting - rreis

yes, and also:

- run with NO openMP, just MPI and serial (-np 1)
- run with NO openMP, just MPI and parallel

no no, that is the whole issue!! my openmp part is running fine, now i want some help from you to make it work across the nodes of a cluster.
i will check your suggestion though

[roddur]

i have checked your suggestion!! the serial run compiled with mpi is running ok(i dont yet know how good is the result; serial run will take days), but for parallel run there is some warnning:
$ mpiexec -np 8 ./irun
Submitted on hpc.bose.res.in date 04.17.2009 22:06:33

Atomic number of Atom A = 84.0000000000000
Atomic number of Atom B = 24.0000000000000
Concentration of Atom A = 0.700000000000000
Concentration of Atom B = 0.300000000000000
Running spin-dependent calculation
Running ASR loop 1
Submitted on hpc.bose.res.in date 04.17.2009 22:06:33
forrtl: No such file or directory
forrtl: severe (29): file not found, unit 1, file /home/abhijit/mpif-rec/ASR/CHECK_A
Image PC Routine Line Source
irun 000000000049E8DF Unknown Unknown Unknown
irun 000000000049CCA6 Unknown Unknown Unknown
irun 00000000004801F4 Unknown Unknown Unknown
irun 000000000044E833 Unknown Unknown Unknown
irun 000000000044E49C Unknown Unknown Unknown
irun 00000000004532E6 Unknown Unknown Unknown
irun 0000000000406F64 Unknown Unknown Unknown
irun 000000000040392A Unknown Unknown Unknown
libc.so.6 0000003A1471C3FB Unknown Unknown Unknown
irun 000000000040386A Unknown Unknown Unknown
forrtl: No such file or directory
forrtl: No such file orSubmitted on hpc.bose.res.in date 04.17.2009 22:06:33
Submitted on hpc.bose.res.in date 04.17.2009 22:06:33
Submitted on hpc.bose.res.in date 04.17.2009 22:06:33
directory
forrtl: severe (29): file not found, unit 1, file /home/abhijit/mpif-rec/ASR/CHECK_A
Image PC Routine Line Source
irun 000000000049E8DF Unknown Unknown Unknown
irun 000000000049CCA6 Unknown Unknown Unknown
irun 00000000004801F4 Unknown Unknown Unknown
irun 000000000044E833 Unknown Unknown Unknown
irun 000000000044E49C Unknown Unknown Unknown
irun 00000000004532E6 Unknown Unknown Unknown
irun 0000000000406F64 Unknown Unknown Unknown
irun 000000000040392A Unknown Unknown Unknown
libc.so.6 0000003A1471C3FB Unknown Unknown Unknown
irun 000000000040386A Unknown Unknown Unknown
forrtl: severe (29): file not found, unit 1, file /home/abhijit/mpif-rec/ASR/CHECK_A
Image PC Routine Line Source
irun 000000000049E8DF Unknown Unknown Unknown
irun 000000000049CCA6 Unknown Unknown Unknown
irun 00000000004801F4 Unknown Unknown Unknown
irun 000000000044E833 Unknown Unknown Unknown
irun 000000000044E49C Unknown Unknown Unknown
irun 00000000004532E6 Unknown Unknown Unknown
irun 0000000000406F64 Unknown Unknown Unknown
irun 000000000040392A Unknown Unknown Unknown
libc.so.6 0000003A1471C3FB Unknown Unknown Unknown
irun 000000000040386A Unknown Unknown Unknown
forrtl: No such file or directory
forrtl: severe (29): file not found, unit 1, file /home/abhijit/mpif-rec/ASR/CHECK_A

and so on which i found strange 'coz CHECK_A is read is serial part and not in mpi region!! should i broadcast that though?

[andrewmcleod]

I'm not sure how you have a 'serial part' - I think the first thing you should do in an MPI code is call MPI_INIT. From the MPICH2 docs:

"The MPI standard does not say what a program can do before an MPI_INIT or after an MPI_FINALIZE. In the MPICH implementation, you should do as little as possible. In particular, avoid anything that changes the external state of the program, such as opening files, reading standard input or writing to standard output."

[TimP]

Quoting - andrewmcleod

I'm not sure how you have a 'serial part' - I think the first thing you should do in an MPI code is call MPI_INIT. From the MPICH2 docs:

"The MPI standard does not say what a program can do before an MPI_INIT or after an MPI_FINALIZE. In the MPICH implementation, you should do as little as possible. In particular, avoid anything that changes the external state of the program, such as opening files, reading standard input or writing to standard output."

Perhaps you have inside information that MPICH2 is meant here, but that doesn't justify ignoring the standard about using MPI_INIT_THREADS rather than MPI_INIT when using OpenMP or other threading. Even if MPI_INIT_THREADS is used, I wouldn't count on a run-time indication on whether you used your threading and MPI in a manner consistent with your MPI_INIT_THREADS call, unless you run with the Intel message checking library.
If you are emphasizing the part of the advice about calling MPI_INIT[_THREADS] and MPI_FINALIZE as early and late as possible, then that may be good advice.

[andrewmcleod]

Sorry, I did only intend to say that MPI should be initialized at the start of the code, whether by MPI_INIT or MPI_INIT_THREADS (which did not occur to me as I have never run a threaded MPI application).

[roddur]

Quoting - tim18

When you enable OpenMP along with MPI, you must change from MPI_INIT to MPI_INIT_THREAD, checking whether the type of threading support you require is enabled.
You would require Intel MPI 3.2 to get affiinity support, engaged by setting I_MPI_PIN_DOMAIN=omp

can you plz be a bit more specific? in most example of hybrid code, i have seen them using MPI_INIT only, on the otherhand,in mpi_thread_init(required,**) the first option is "Level of desired thread support" what that is?

[TimP]

Quoting - roddur

can you plz be a bit more specific? in most example of hybrid code, i have seen them using MPI_INIT only, on the otherhand,in mpi_thread_init(required,**) the first option is "Level of desired thread support" what that is?

If you want assurance that MPI_INIT_THREAD isn't an invention of Intel MPI, look at public man pages, such as
http://linux.die.net/man/3/mpi_init_thread
I understand that HP-MPI may work for the MPI_THREAD_FUNNELED case when MPI_INIT is used instead of MPI_INIT_THREAD, but that is an extension to the MPI standard. If an MPI doesn't actually implement MPI_INIT_THREAD or any threading support, it may still be possible to make limited use of a hybrid model, for example, with 1 MPI process per node. That may often be an acceptable limitation.
This is getting more into the territory of the HPC forum, given that the compilers which are topical here don't include their own MPI.