Issue with Running mpirun inside docker container

[carlosgar]

I am trying to run the Linpack benchmark found with under here: https://software.intel.com/content/www/us/en/develop/articles/intel-mkl-benchmarks-suite.html

The Linpack benchmark uses mpirun and I get the following error:

[root@7b514c24d78f mp_linpack]# ./runme_intel64_dynamic -p 2 -q 1 -b 384 -n 80000
This is a SAMPLE run script.  Change it to reflect the correct number
of CPUs/threads, number of nodes, MPI processes per node, etc..
This run was done on: Mon Oct  4 18:19:23 UTC 2021
RANK=1, NODE=1
RANK=0, NODE=0
./runme_intel64_prv: line 30:   890 Bus error               (core dumped) ./${HPL_EXE} "$@"

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 888 RUNNING AT 7b514c24d78f
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

 We believe the error is with mpirun since the following line gives a similar issue:

[root@f9103950d892]# mpirun -np 2 IMB-MPI1

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 0 PID 2669 RUNNING AT f9103950d892
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 2670 RUNNING AT f9103950d892
=   KILLED BY SIGNAL: 7 (Bus error)
===================================================================================

This is all being run inside a docker container. I think there is an issue with running mpirun inside a docker container, is there a way to work around this?

[SantoshY_Intel]

Hi,

Thanks for reaching out to us.

Could you please provide us the following details to investigate more on your issue?

1. The version of the operating system being used.

2. The version of intel oneAPI installed.

3. The usage model that you have followed to run the application from the link:

https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/running-applications/running-intel-mpi-library-in-containers/run-the-application-with-a-container.html

4. The steps to reproduce your issue.

Thanks & regards,

Santosh

[carlosgar]

1. CentOS 8

This is the Dockerfile I built which answers the other questions:

FROM centos:8.3.2011 AS build

RUN dnf -y install epel-release && dnf group -y install "Development Tools" && dnf -y install wget cmake libarchive

RUN no_proxy=$(echo $no_proxy | tr ',' '\n' | grep -v -E '^\.?intel.com$' | tr '\n' ',') yum install intel-hpckit -y

RUN wget https://software.intel.com/content/dam/develop/external/us/en/documents/l_onemklbench_p_2021.2.0_109.tgz && tar -xvzf l_onemklbench_p_2021.2.0_109.tgz
RUN dnf install numactl -y

These are the steps I take to run the benchmark that uses mpirun:

cd benchmarks_2021.2.0/linux/mkl/benchmarks/mp_linpack/
source /opt/intel/oneapi/setvars.sh
./runme_intel64_dynamic -p 2 -q 1 -b 384 -n 80000

 Not sure what you mean about the usage model. Isn't the latest version of mpi included in the latest oneAPI hpc toolkit?

[SantoshY_Intel]

Hi,

Thanks for providing the details.

>>" Not sure what you mean about the usage model."

     You can choose from three usage models for running your application using a singularity container. For more information, you can refer to the below link.
https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/running-applications/running-intel-mpi-library-in-containers/run-the-application-with-a-container.html

 >>"Isn't the latest version of mpi included in the latest oneAPI hpc toolkit?"

     Yes, the latest oneAPI HPC toolkit includes the latest version of Intel MPI Library.

We are working on your issue and we will get back to you soon.

Thanks & Regards,

Santosh

[SantoshY_Intel]

Hi,

We are able to reproduce your issue at our end. However, we tried to run the Linpack benchmark successfully using the below workaround on a ubuntu 18.04 machine:

1. The docker file that we built on the ubuntu machine is given below:
   

   FROM tacc/tacc-ubuntu18-impi19.0.7-common:latest
   
   RUN apt-get update && apt-get upgrade -y && apt-get install wget
   
   RUN wget --user-agent="Mozilla" https://www.intel.com/content/dam/develop/external/us/en/documents/l_onemklbench_p_2021.2.0_109.tgz
   RUN tar -xvzf l_onemklbench_p_2021.2.0_109.tgz​

2. To build a new MPI-capable container, use the below command:
   

    docker build -t USERNAME/pi-estimator:0.1-mpi -f Dockerfile.mpi .​

   
   

   Note: Don’t forget to change USERNAME to your DockerHub username!

3. To push the successfully built image to the DockerHUb, use the below command:
   

   docker push USERNAME/pi-estimator:0.1-mpi​

4. To run the Linpack benchmark, use the below command:
   

    docker run --rm -it USERNAME/pi-estimator:0.1-mpi \
           benchmarks_2021.2.0/linux/mkl/benchmarks/mp_linpack/runme_intel64_dynamic -p 2 -q 1 -b 384 -n 80000​

Could you please try the above workaround using "Intel MPI Library 2019 update 7" and let us know the outcomes?

Meanwhile, we will get back to you regarding the error while using Intel oneAPI HPC Toolkit 2021.4.

Thanks & regards,

Santosh

[SantoshY_Intel]

Hi,

We haven't heard back from you. Have you tried the above workaround? Please get back to us if you face any issues while following the workaround provided above.

Thanks & Regards,

Santosh

[SantoshY_Intel · Accepted solution]

Hi,

>>"Meanwhile, we will get back to you regarding the error while using Intel oneAPI HPC Toolkit 2021.4."

We are able to run IMB-MPI1 and the Linpack benchmark successfully using the latest Intel oneAPI 2021.4 as well.

We followed the below steps:

1. docker pull intel/oneapi-hpckit
2. docker run --shm-size=4gb -it intel/oneapi-hpckit
3. cd /opt/intel/oneapi/mkl/latest/benchmarks/mp_linpack
4. ./runme_intel64_dynamic -p 2 -q 1 -b 384 -n 80000

In brief, adding the "--shm-size=4gb" option to the "docker run" statement resolves this issue.

If this resolves your issue, make sure to accept this as a solution. This would help others with similar issues.

Thank you!

Best Regards,

Santosh

[SantoshY_Intel]

Hi,

Thanks for accepting our solution. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Thanks & Regards,

Santosh