600 permissions (on Linux) for log files from mpirun

[vipulk]

Hi,

I actually have 2 questions on mpirun usage. 

1. I would like MPI application log file to be such that both stdout and stderr from them goes in the same file. What is the best way to achieve that? I am currently using below command (basically specifyin same pattern for both outfile and errfile) but wanted to understand if there is different recommended way.

mpirun -n <N> -outfile-pattern application_%r.log -errfile-patter application_%r.log

2. The application log files with this method are created with 600 permission for users (on Linux), so only read permission for owner. Is there a way to control this behavior? I am sure its not impacted by umask settings, since I have it in my environment and it does not seem to honor it.

I tried to look up an answer for these question but could not find a related answer.

Thanks

Vipul

[AbhishekD_Intel]

Hi Vipul,

Mostly there are only the requirements to analyze the stdout and stderr so the flags( -outfile-pattern and -errfile-patter ) will help to generate them, but if you want to collect all of these logs into a single log file, then you can try rerouting your all outputs into a single file. The below command will help you to do so.

mpirun -n <num> ./<executable> 2>&1 | tee <output_log>

By default, all of these log files will get created with 600 permission i.e. read and write for the users. And we think this permission is sufficient for log files, but if you want then you can change them manually with a command like chmod <permission> <filename>.

Hope these details will help you to solve your use-case.

Warm Regards,

Abhishek

[vipulk]

Thanks Abhishek.

Maybe you have misunderstood my question. I would still like the log files to be created on per rank basis. However, I would like to both stdout and stderr for each process to go in rank - wise log file. Currently I am achieving that by specifying EXACTLY same pattern in both -outfile-pattern and -errfile-pattern, but was wondering if there is different recommended method. The suggested command will cause the stdout/stderr for all ranks to go into the output log file.

For the permissions, my question is whether there is an option in mpirun to generate the files with desired permissions. If not, then I won' have an option but to do a 'chmod'.

Thanks,

Vipul

[AbhishekD_Intel]

Hi Vipul,

It seems that you want a rank-wise stdout and stderr logs in a single file with rank id related to particular stdout/stderr. If this is the case then you can use a -prepend-rank flag with your mpirun command and redirect the whole output into a single file. You will get the rank inside square brackets to identify the stdout/stderr associated with that rank.

Please refer the following command:

mpirun -n 6 -prepend-rank ./hello 2>&1 | tee file

This is the way we used to maintain and read collective logs from different ranks. Let us know if this works for your use-case.

And I think there is no such specific flag or environment variable to change the permission of the log file with mpirun. So, in this case, you can go with chmod.

Warm Regards,

Abhishek

[vipulk]

HI Abhishek,

No I do not want a single log file with output from all ranks.

I want one log file for each rank and I would like both stdout and stderr from a given rank to go to this rank specific log file.

For example, if I had 3 ranks: 

I would rank0.log, rank1.log and rank2.log files. The rank0.log should contain both stdout and stderr from rank 0 process. rank1.log file should contain both stdout and stderr from rank 1 process. rank2.log file should contain both stdout and stderr from rank2 process.

I am getting this desired behavior by using the command

% mpirun -outfile-pattern rank_%r.log -errfile-pattern rank_%r.log -n <3>...

My question is if there is a different recommended method to get this behavior.

Thanks,

Vipul

[AbhishekD_Intel · Accepted solution]

Hi Vipul,

Thanks for the example. I was interpreting it differently. It seems that you are following the correct way to collect the logs. And I haven't seen any other particular flag to collect all these logs for different ranks.

The way you have suggested (mpirun -n <N> -outfile-pattern rank_%r.log -errfile-pattern rank_%r.log ) is the correct one.

Thank You,

Abhishek

[AbhishekD_Intel]

Hi Vipul,

Thanks for the update. Please post a new thread if you have any other issues.

Warm Regards,

Abhishek