Where is the /Qcoarray:distributed option?

[obmeninfor]

Hi All,

I have installed the Intel Cluster Studio XE 2012 for Windows (file "w_ics_2012.0.033.exe") using Evaluation license file received from Intel, but I cant evaluate the cluster work of Fortran in the Properties of my new project (RClick --> Properties --> Configuration Properties --> Fortran --> Language --> Enable Coarrays) I don't see the option for Distributed Memory (/Qcoarray:distributed) only "No" and "For Shared Memory (/Qcoarray:shared)" for both Win32 and x64 solution platforms.

My cluster system consists of 2 computers:
1) Head node: Windows Server 2008 R2 with SP1 + HPC Pack 2008 R2 with SP3 + Visual Studio 2010 with SP1;
2) Workstation node: Windows 7 (x64) with SP1 + HPC Pack 2008 R2 with SP3.

The Intel Cluster Studio was being installed on the head node, but automatically it was installed on the workstation node too.

If I insert the /Qcoarray:distributed option manually (RClick --> Properties --> Configuration Properties --> Fortran --> Command Line --> Additional Options: /Qcoarray:distributed), a test program works on the head node only, although the corresponding machines.Windows file (the environment (system) variable FOR_COARRAY_MACHINEFILE is assigned) has 2 lines with the computer node names.

The result of command "clusrun smpd status" is

----- Summary ----
2 Nodes succeeded
0 Nodes failed

What is wrong and what should I do to see the "/Qcoarray:distributed" option?

[James_T_Intel]

Hi obmeninfor,

What is the output from "smpd -get binary" on each of the computers?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[obmeninfor]

Hi James,

1) smpd get binary on WinSer2008R2:
C:\Program Files (x86)\Intel\MPI\4.0.3.009\em64t\bin\smpd.exe
2) smpd get binary on Win7:
C:\Program Files (x86)\Intel\icsxe\2012.0.033\mpi\em64t\bin\smpd.exe

What is about the log files?

Thank you.

[James_T_Intel]

Hi obmeninfor,

I've attachedthe smpd log files from one of my runs if you are interested. What appears to be happening is that the processes are either not launching correctly or not attaching to the smpd correctly. In the logfile, you should see at the beginning of each line the rank and PID of the process involved in the event in the form [rank:PID]. When the rank is -1, the actual processes have not yet been started, the messages are related to the smpd itself.

On your Windows* 7 log, there are no processes started within the smpd. On your Windows* Server 2008, only 1 process is started, and it is incorrectly numbered (C numbering, the first process should be 00, not 01). With the PID changing in the Windows* Server 2008 log, it appears that the processes are not starting as a group, but as individual processes started simultaneously, but expecting a group.

Additionally, the Windows* Server 2008 log appears to be corrupted (see lines 371, 401, 404, and 416 for some examples).

Can you run the following commands on each computer?

[plain]smpd -V
smpd -version
mpiexec -V[/plain]

I'm hoping to find a version mismatch that could be the cause of the problem, though it seems very odd that a standard MPI program would run and a coarray (using MPI) would not.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[obmeninfor]

Hi James,

Thank you for your analysis of the log files. It is really strangely as I wrote above, the coarray program runs and works but does not close.

On WinSer2008R2:

1)smpd V
Intel MPI Library for Windows* OS, Version 4.0 Update 2 Build 4/28/2011 6:04:28 PM
Copyright (C) 2007-2011, Intel Corporation. All rights reserved.

2)smpd version
3.1

3)mpiexec V
Intel MPI Library for Windows* OS, Version 4.0 Update 2 Build 4/28/2011 6:04:28 PM
Copyright (C) 2007-2011, Intel Corporation. All rights reserved.

On Win7:

1)smpd V
Intel MPI Library for Windows* OS, Version 4.0 Update 2 Build 4/28/2011 6:04:28 PM
Copyright (C) 2007-2011, Intel Corporation. All rights reserved.

2)smpd version
3.1

3)mpiexec V
Intel MPI Library for Windows* OS, Version 4.0 Update 2 Build 4/28/2011 6:04:28 PM
Copyright (C) 2007-2011, Intel Corporation. All rights reserved.


Any ideas?

Thanks

[James_T_Intel]

Hi obmeninfor,

Everything appears to match up between the two systems. Let's try one more experiment. Modifyhello_image.f90 to this:

[fortran]      program hello_image

      character*256 hostname
      character*512 outstr
      integer length, status

      call get_environment_variable("COMPUTERNAME",hostanme,length,status,.true.)

      write(outsr,*) "Image ",this_image()," of ", num_images()," on ",trim(hostname)
      write(*,*) trim(outstr)

      end program hello_image
[/fortran]

Try running this program and let me know what happens. As I said earlier, it looks like the processes are not being properly distributed, and this will show where each process is really running.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[obmeninfor]

Hi James,

Your code prints 8 lines, - and hangs:

 Image 5 of 8 on WinSer2008R2
 Image 3 of 8 on WinSer2008R2
 Image 7 of 8 on WinSer2008R2
 Image 1 of 8 on WinSer2008R2
 Image 8 of 8 on Win7
 Image 2 of 8 on Win7
 Image 6 of 8 on Win7
 Image 4 of 8 on Win7

Ctrl^C prints additional lines:

mpiexec aborting job...

job aborted:
rank: node: exit code[: error message]
0: WinSer2008R2.mynet.dom: 123: mpiexec aborting job
1: Win7: 123
2: WinSer2008R2.mynet.dom: 123
3: Win7: 123
4: WinSer2008R2.mynet.dom: 123
5: Win7: 123
6: WinSer2008R2.mynet.dom: 123
7: Win7: 123

Thanks.

[James_T_Intel]

Hi obmeninfor,

Would it be possible to try running the executable from a non-shared location? Use the same path on each computer, and have a copy of the executable in each folder.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel Cluster Tools

[obmeninfor]

Hi James,

I have done in accordance with your advice (setthe project Output File option: C:\$(ProjectName).exe, and copied thecoarray_samples.exeinto C:\ on Win7)- the result is the same.

Thanks

[Lorri_M_Intel]

Hi --

I told you these guys were awesome; I'm glad to see they got a straight-MPI program running for you.

Anyway - since the problem now seems to be strictly coarray-related, I'm back.

What version of Intel Fortran do you have? From the command window, would you issue the following command, and let me know what comes back?

ifort -what

This should return the full logo-banner, and our internal edit number, something similar to this:

F:\tests>ifort -what end.f -c

Intel Visual Fortran Compiler XE for applications running on IA-32, Version 12.0.2.154 Build 20110112

Copyright (C) 1985-2011 Intel Corporation. All rights reserved.

Intel Visual Fortran 12.0-1259


We found some reports of coarrays hanging in our internal database, but they were fixed/available about a year ago. The "Build" date of the full logo-banner would let us know that we weren't chasing known problems.

There are a couple of other things we can try after we're sure it's not one of these older problems.

Again -- thank you for using the Windows DCAF!

-- Lorri

[obmeninfor]

Hi Lorri,

I have installedw_ics_2012.0.033.exe.

Command ifort -what hello_image.f90 c prints:

Intel Visual Fortran Intel 64 Compiler XE for applications running on Intel 64,Version 12.1.0.233 Build 20110811
Copyright (C) 19852011 Intel Corporation. All rights reserved.

Intel Fortran 12.1-2054

Thanks.

[Han_T_]

Hi, everyone

For those of you who still don't know where to find the corresponding Coarray options under VS2017:

After you open the fortran project, for example, filename.f90.

Go to Project => filename Properties...  => Fortran => Language => Enable Coarrays (For Shared Memory), Coarray Images 8

That should work. Good luck :)

-- Han