MPI Fortran Program in PSXE 2019 Cluster Edition in VS2017 Runs each node as master node

[Bilal_B_]

I am having a strange issue with all updates of PSXE2019 cluster edition (Windows) where all nodes run as master nodes. There are no issues with PSXE2018 or prior versions.Starting with Update 1 last November, I have reinstalled PSXE2019 many times in the hope that it will resolve the issue. As a last measure, today I uninstalled all previous updates and Installed Update 3 of PSXE109, but unfortunately I have the same problem. For example, When I run the sample test.f (a simple "helloworld" program that comes with the installation) on four nodes  (-n 4) I will get four independent master images running, each image shows the total number of available nodes as 1 (RANK=0 and SIZE=1). If I uninstall 2019and revert back to 2018 version, it will run fine and each image will show a correct count of 4 total images (RANK= 0 to 3 and SIZE=4).

Again, I have no problems with all updates of the prior PSXE2018 cluster edition. 

I have asked Intel Support but they have not been able to replicate my problem. I am wondering if anyone else has had the same or similar problem, or if someone can give me some ideas as to what is going on. Thanks

Some specifics:

1. Operating System: Windows 10 Pro
2. Workstation: 44 Core Dual-Xeon Processor Desktop  
3. Installed PSXE 2019 Cluster Edition (Update 3) using the downloaded installation file: parallel_studio_xe_2019_update3_cluster_edition_setup.exe (3,130,849 KB)
4. Configured Visual Studios 2017 as per instruction from document “Configuring a Visual Studio* Project” downloaded from https://software.intel.com/en-us/mpi-developer-guide-windows-configuring-a-visual-studio-project
5. Using the configuration instructions, my compiler command line is: /nologo /debug:full /Od /I"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.3.203\windows\mpi\intel64\include" /warn:interfaces /module:"x64\Debug\\" /object:"x64\Debug\\" /Fd"x64\Debug\vc150.pdb" /traceback /check:bounds /check:stack /libs:dll /threads /dbglibs /c
6. My Linker Command line is: /OUT:"x64\Debug\test_mpi.exe" /INCREMENTAL:NO /NOLOGO /LIBPATH:"C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.3.203\windows\mpi\intel64\lib\debug" /MANIFEST /MANIFESTFILE:"x64\Debug\test_mpi.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG /PDB:"C:\My_VFXePrograms\test_mpi\test_mpi\x64\Debug\test_mpi.pdb" /SUBSYSTEM:CONSOLE /IMPLIB:"C:\My_VFXePrograms\test_mpi\test_mpi\x64\Debug\test_mpi.lib" impi.lib
7. I used 4 nodes (-n 4) to compile and run test.f (C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.3.203\windows\mpi\intel64\test). The resulting output of the test.f code is shown in the attached slide
8. On each node the call MPI_COMM_SIZE (MPI_COMM_WORLD, size, ierr) command is returning size=1 instead of 4 
9. On each node the call MPI_COMM_RANK (MPI_COMM_WORLD, rank, ierr) command is returning RANK=0 instead of 4 unique values 0, 1, 2 and 3, respectively

[Steve_Lionel]

I don't profess to be an expert in this area, nor do I have much experience with MPI at all, but the first thing I suggest is to check the value of PATH to see which (and how many) location(s) of the Intel MPI runtime are listed there. When you install Fortran, it installs an MPIRT for use with the coarray feature, and the Cluster Edition has its own copy of Intel MPI (possibly newer, possibly older) that it installs. You'll want to see which MPIRT triggers the problem.

[TimP · Accepted solution]

As Steve said, ifort includes a version of MPI which supports only co-arrays.  The Cluster Edition, which has a separately installable full MPI implementation, doesn't change this.  If you intend to access MPI from a command line, the supported way to add the full MPI to PATH is to run the mpivars.bat in the MPI installation.  If you are building an MPI application in the GUI, you must add the MPI library specifically to the library paths there.  I don't have the current version, but in the past there has been a GettingStarted text file in the MPI installation which is important to read.

[Bilal_B_]

Hi Steve & Kim,

Thank you for your suggestion about checking the path and environment variables. I had incorrectly assumed that the installation & un-installation procedures will correctly set those up.  Somehow the system PATH variable  pointed to not only the mpi folder under PSXE2019.3.203 but also the prior 2018.3.210 and 2018.5.274 folders. Apparently, when upgrading from the previous version, it never fully got rid of there folders and path information.

To fix the issue, I did the following:

1. First manually uninstalled all Parallel Studio XE versions (2018.3.210 & 2018.5.274 & 2019.3.203).
2. Then edit the system PATH variable (Control Panel>System and Security>System>Advanced system settings>Environment Variables) and removed any left over mention of these versions in the system PATH  variable (in my case 2018.3.210 was still in the path).
3. Then install the latest PSXE2019.3.203  update.

Thank you again for pointing me in the right direction.

Best regards,

Bilal  

[Steve_Lionel]

Yeah, it's been a general annoyance that the PSXE installers tend to clutter up PATH. Glad to hear you got it sorted.