Parallel CFD simulations with OpenFOAM

[Kirils]

Hello! I am trying to run OpenFOAM-6 simulations on the Intel DevCloud cluster. 

I compiled OpenFOAM-6 as described here, and used module load mpi/2021.7.1, the installation was successful. After the compilation, the serial (one core) simulation works fine. However, when I run the parallel simulations with mpirun, the following errors appear. 

$ qsub -l nodes=1:ppn=2 -I

$ module load mpi/2021.7.1

$ which mpirun
/glob/development-tools/versions/oneapi/2023.0/oneapi/mpi/2021.8.0/bin/mpirun

$ which icoFoam
/home/u183499/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/bin/icoFoam

$ mpirun -np 2 icoFoam -parallel
-------------------------------------------------------------------
WARNING: There is at least non-excluded one OpenFabrics device found,
but there are no active ports detected (or Open MPI was unable to use
them). This is most certainly not what you wanted. Check your
cables, subnet manager configuration, etc. The openib BTL will be
ignored for this job.

Local host: s001-n059
--------------------------------------------------------------------
[s001-n059:1507980] *** Process received signal ***
[s001-n059:1507980] Signal: Segmentation fault (11)
[s001-n059:1507980] Signal code: Address not mapped (1)
[s001-n059:1507980] Failing at address: 0x440000e8
[s001-n059:1507980] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f6a8522a090]
[s001-n059:1507980] [ 1] /lib/x86_64-linux-gnu/libmpi.so.40(MPI_Comm_rank+0x3b)[0x7f6a84ebbc2b]
[s001-n059:1507980] [ 2] /home/u183499/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/mpi-system/libPstream.so(_ZN4Foam8UPstream4initERiRPPcb+0x2d)[0x7f6a851de25d]
[s001-n059:1507980] [ 3] /home/u183499/OpenFOAM/OpenFOAM-6/platforms/linux64GccDPInt32Opt/lib/libOpenFOAM.so(_ZN4Foam7argListC1ERiRPPcbbb+0x5a0)[0x7f6a85aa0f50]
[s001-n059:1507980] [ 4] icoFoam(+0x25af8)[0x556268e63af8]
[s001-n059:1507980] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f6a8520b083]
[s001-n059:1507980] [ 6] icoFoam(+0x293ae)[0x556268e673ae]
[s001-n059:1507980] *** End of error message ***
[s001-n059:1507982] PMIX ERROR: NO-PERMISSIONS in file ../../../../../../src/mca/common/dstore/dstore_base.c at line 234
[s001-n059:1507982] PMIX ERROR: NO-PERMISSIONS in file ../../../../../../src/mca/common/dstore/dstore_base.c at line 243

In order to debug, I tried a simple MPI hello-world program, and it worked well:

$ cat hello.cpp
#include <mpi.h>
#include <stdio.h>
int main(int argc, char** argv) {
// Initialize the MPI environment
MPI_Init(NULL, NULL);

// Get the number of processes
int world_size;
MPI_Comm_size(MPI_COMM_WORLD, &world_size);

// Get the rank of the process
int world_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);

// Get the name of the processor
char processor_name[MPI_MAX_PROCESSOR_NAME];
int name_len;
MPI_Get_processor_name(processor_name, &name_len);

// Print off a hello world message
printf("Hello world from processor %s, rank %d out of %d processors\n",
processor_name, world_rank, world_size);

// Finalize the MPI environment.
MPI_Finalize();
}

$ mpicc hello.cpp -o hello.exe

$ mpirun -n 2 ./hello.exe
Hello world from processor s001-n059, rank 0 out of 2 processors
Hello world from processor s001-n059, rank 1 out of 2 processors

Do you have any idea, how could I make my OpenFOAM installation work with mpirun?

Best regards,

Kirils

[VarshaS_Intel]

Hi,

Thanks for posting in Intel Communities.

Thanks for the information.

From the link provided by you for OpenFOAM-6(https://openfoam.org/download/6-source/), we have seen that it is supported/tested till Ubuntu 18.04. In Intel Devcloud the ubuntu version is 20.04.

Could you please try the supported/tested OpenFOAM version? If your issue persists, please let us know the complete steps you followed while using/compiling the Intel MPI with OpenFOAM.

>>After the compilation, the serial (one core) simulation works fine.

And also, could you please provide the complete steps and expected output while running on a single node?

Thanks & Regards,

Varsha 

[Kirils]

Good afternoon, and thank you for the suggestion!

I tried to install the newer OpenFOAM version, which was tested on Ubuntu 20.04. I followed the steps from https://openfoam.org/download/10-source/, all these commands are shown below:

qsub -I
mkdir OpenFOAM
cd OpenFOAM
wget -O - http://dl.openfoam.org/source/10 | tar xz
wget -O - http://dl.openfoam.org/third-party/10 | tar xz
mv OpenFOAM-10-version-10 OpenFOAM-10
mv ThirdParty-10-version-10 ThirdParty-10
cd OpenFOAM-10
source etc/bashrc

There was an error, therefore I changed the MPI version on line 89 of file etc/bashrc from SYSTEMOPENMPI to SYSTEMMPI, then I loaded it with module and tried again:

module load mpi/2021.7.1
source etc/bashrc

There is another error, according to its text I added the following environment variables, sourced etc/bashrc again and ran the compilation:

export MPI_ROOT=/glob/development-tools/versions/oneapi/2023.0/oneapi/mpi/2021.8.0
export MPI_ARCH_FLAGS="-DOMPI_SKIP_MPICXX"
export MPI_ARCH_INC="-isystem $MPI_ROOT/include"
export MPI_ARCH_LIBS="-L$MPI_ROOT/lib -lmpi"
source etc/bashrc
./Allwmake -j >log 2>&1 &

However, I got these errors in the log file:

y.tab.c:270:6: error: conflicting types for ‘scotchyyerror’; have ‘void(const char *)’
parser_yy.c:70:25: note: previous declaration of ‘scotchyyerror’ with type ‘int(const char * const)’
70 | #define yyerror scotchyyerror
| ^~~~~~~~~~~~~
parser_yy.h:83:29: note: in expansion of macro ‘yyerror’
83 | static int yyerror (const char * const);
| ^~~~~~~
parser_yy.c:70:25: error: conflicting types for ‘scotchyyerror’; have ‘int(const char * const)’
70 | #define yyerror scotchyyerror
| ^~~~~~~~~~~~~
parser_yy.y:814:1: note: in expansion of macro ‘yyerror’
814 | yyerror (
| ^~~~~~~
y.tab.c:270:6: note: previous declaration of ‘scotchyyerror’ with type ‘void(const char *)’
make[2]: *** [Makefile:50: parser_yy.o] Error 1

scotchDecomp.C:36:10: fatal error: scotch.h: No such file or directory
36 | #include "scotch.h"

It means that the scotch library from ThirdParty-10 did not compile successfully. Do you have any ideas about how to solve it? I changed the line 65 in etc/bashrc to use the Intel compiler (because I think that it was possible to compile scotch with icx), however the scotch compilation during the execution of ./Allwmake still used gcc despite my changes.

[VarshaS_Intel]

Hi,

Thanks for providing the information.

Could you please use the below link where we can find the different libraries that need to be installed and the lines added in the particular files to link Intel compilers?

https://www.intel.com/content/www/us/en/developer/articles/technical/recipe-compile-openfoam-with-intel-compiler-for-intel-xeon-scalable-processors.html

Could you please try and let us know if you are facing any issues?

Thanks & Regards,

Varsha

[VarshaS_Intel]

Hi,

We have not heard back from you. Could you please provide us with an update on your issue?

Thanks & Regards,

Varsha

[Kirils]

I'm sorry, I had to do another project for a while. Thank you for the provided instructions! However, there are some imprecisions:

1. In Build Directions - Step 1 - MPI_ROOT=/opt/intel/compilers_and_libraries_2018.0.128/linux/mpi/intel64  does not exist, and to proceed I used instead /glob/development-tools/versions/oneapi/2023.0.1/oneapi/mpi/2021.8.0

2. In Build Directions - Step 2 - /opt/intel/compilers_and_libraries_2018.0.128/linux/bin/compilervars.sh does not exist, so I assumed that it is already sourced on the Intel DevCloud cluster and omitted these steps. It seemed to work for the next steps.

3. In Build Directions - Step 3 - Everything was well, except at the end - Building CGAL - I got the following error:

u183499@s001-n015:~/OpenFOAM/ThirdParty-4.1$ ./makeCGAL CGAL-4.8 boost_1_55_0 gmp-5.1.2 mpfr-3.1.2

...
icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
icc.jam: No such file or directory
/home/u183499/OpenFOAM/ThirdParty-4.1/boost_1_55_0/tools/build/v2/build/toolset.jam:41: in toolset.using
ERROR: rule "icc.init" unknown in module "toolset".
/home/u183499/OpenFOAM/ThirdParty-4.1/boost_1_55_0/tools/build/v2/build-system.jam:461: in process-explicit-toolset-requests
/home/u183499/OpenFOAM/ThirdParty-4.1/boost_1_55_0/tools/build/v2/build-system.jam:527: in load
/home/u183499/OpenFOAM/ThirdParty-4.1/boost_1_55_0/tools/build/v2/kernel/modules.jam:289: in import
/home/u183499/OpenFOAM/ThirdParty-4.1/boost_1_55_0/tools/build/v2/kernel/bootstrap.jam:139: in boost-build
/home/u183499/OpenFOAM/ThirdParty-4.1/boost_1_55_0/boost-build.jam:17: in module scope
Error building: boost

[VarshaS_Intel]

Hi,

Thanks for providing the information.

I am facing different errors while running. So, could you please zip the complete folder(all the files related to OpenFOAM)so that we both will be in the same phase and can investigate more on the issue?

Thanks & Regards,

Varsha

[Kirils]

Thank you for the involvement! I created the archive of the whole OpenFOAM directory, which was located in the $HOME directory.

However, its size is too large for this forum, so I put it on the file storage of my university. I hope this link will work: https://store.lu.lv/?s=download&token=2bc0e271-e16d-4791-a477-05c1afb7e20c

In the meantime, I will maybe try to install the newest OpenFOAM-10  version (as in my post from 01-20-2023, because basically any OpenFOAM will work for me, the version number is not important), but using the compiler and other tools from the Intel Recipe that you provided.

With regards,
Kirils

[VarshaS_Intel]

Hi,

Thanks for your efforts in sharing the file.

We are working on your issue. We will get back to you soon.

Thanks & Regards,

Varsha

[VarshaS_Intel]

Hi,

Thanks for your patience.

Apologies for the delayed response. Since the article is outdated and the version of the old OpenFOAM with the latest Intel compiler throws errors.

Could you please try the steps mentioned in the attached text file(OpenFOAM.txt)?

Could you please try and let us know if you are facing any issues?

Thanks and Regards,

Varsha

[Kirils]

Thank you for the detailed instructions! The delay is not a problem, this work is not time-sensitive.

I tried to follow the instructions, and everything worked well until the step 5. OpenFOAM Allwmake command gives me the following error (see the log file in the attachment).

Should I try to install your mentioned packages (zlib1g, zlib1g-dev -y and flex) myself or are they available now as modules or elsewhere on the cluster?

With regards,

Kirils

[VarshaS_Intel]

Hi,

Sorry for the inconvenience, User's don't have admin privileges in Intel DevCloud.

Intel DevCloud is a shared environment that comes with pre-installed validated Intel oneAPI software and complimentary packages.  As a policy, we do not install custom (open source or 3rd party licensed) software in the environment.

We have also tried on Intel Devcloud to build OpenFOAM but we are also facing dependency errors so we tried on other machines and we are able to run it successfully with Intel Compilers.

So, could you please try on other machines and let us know if you are facing any errors?

Thanks & Regards,

Varsha

[Kirils]

Good morning!

Ok, I understand the policy. Fortunately, I did another thing successfully: I installed (in the $HOME of my user on DevCloud) OpenFOAM-v2212 with your instructions, the only difference being gcc instead of Intel compiler. And then I also installed bison and flex, according to compilation errors. Then it was necessary to add these changes to $PATH:

export PATH=$PATH:/home/u183499/soft/bison-3.8/build/bin
export PATH=$PATH:/home/u183499/soft/flex-2.6.4/build/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/u183499/soft/flex-2.6.4/build/lib
export CPLUS_INCLUDE_PATH=/home/u183499/soft/flex-2.6.4/build/include
export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/home/u183499/OpenFOAM/ThirdParty-10/scotch_6.0.9/include

After that, the OpenFOAM-v2212/tutorials/incompressible/icoFoam/cavity case worked both serially and in parallel! Therefore, my issue is solved and the thing that I wanted is working, I think the thread can be closed.

Thank you for the assistance!

[VarshaS_Intel]

Hi Kirils,

>>I wanted is working, I think the thread can be closed

Thanks for the confirmation. Glad to know that your issue is resolved. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel.

Have a Good Day!

Thanks & Regards,

Varsha