topic Dear Jim, in Intel® Moderncode for Parallel Architectures

split output by mpi rank

Timo_W_ — Thu, 26 Oct 2017 06:45:47 GMT

Hi all,

I was wondering if there is a possiblity to redirect and split stdout of each mpi rank to a seperate file, so that I can investigate the output of each mpi rank seperately.

best regards,

Timo

Is your program written in C

jimdempseyatthecove — Thu, 26 Oct 2017 17:17:00 GMT

Is your program written in C/C++ or Fortran. Both can replace stdout with a file (i.e file with rank number embedded).

https://stackoverflow.com/questions/18086193/redirect-stdout-stderr-to-file-under-unix-c-again

Jim Dempsey

Dear Jim,

Timo_W_ — Fri, 27 Oct 2017 11:17:39 GMT

Dear Jim,

it is written in Fortran

best regards

Timo

Fortran has interoperablee

jimdempseyatthecove — Fri, 27 Oct 2017 16:52:42 GMT

Fortran has interoperablee function calls. You can call C from Fortran, and link in an object file generated by C/C++. This is relatively easy to do to get what you want.

Jim Dempsey

I don't know if it is

McCalpinJohn — Mon, 30 Oct 2017 12:21:25 GMT

I don't know if it is possible to do this directly with a command of the form: "mpirun mpi_binary.exe", but it should be easy enough to do if you add a "wrapper" script in the middle.

Most of my MPI jobs are launched with commands of the form: "mpirun wrapper.sh mpi_binary.exe".

In this case, mpirun launches one instance of "wrapper.sh" for each MPI rank. In the wrapper.sh script, I set up output file names based on the hostname and MPI rank number, then execute the "mpi_binary.exe" with rank-specific IO redirection.

A simple example might be (untested):

#!/bin/bash

MYHOST=`hostname -s`
MYRANK=$PMI_RANK           # this variable may depend on your execution environment

PERFOUT=perf.$MYHOST.$MYRANK
STD_OUT=stdout.$MYHOST.$MYRANK
STD_ERR=stderr.$MYHOST.$MYRANK

# assume that the arguments to this script are the name of the executable
# followed by any of its required options

perf stat -o $PERFOUT $* 2> $STD_ERR 1> $STD_OUT

Dear John,

Timo_W_ — Mon, 30 Oct 2017 13:15:24 GMT

Dear John,

this might help me. I will investigate the example and try to adapt it to my case. Thanks!

best,

Timo