Breakpoints don't work in a coarray program

[Jon_D]

Hello,

I am trying out the co-arrays but found out that I can't debug a program with co-arrays. Specifically, the debugger does not stop at breakpoints. I am using compiler version 17.0.8.275 (latest version 19.0.0.117 has the same issue as well) with Visual Studio 2017 Community Edition. Are there any flags in VS that I need to set to be able to use the the debugger? Any help will greatly be appreciated.

Thanks,

Jon

[Steve_Lionel]

https://software.intel.com/en-us/articles/how-to-debug-fortran-coarray-applications-on-windows

[Jon_D]

Hi Steve,

I had found this article and tried the suggestions. I place a breakpoint as you suggest in the article but the debugger never stops at the breakpoint.

[Steve_Lionel]

Well, it worked when I wrote the article! I will try it myself tomorrow and see what I find.

[Steve_Lionel]

I just tried it with 19.0.1, following the steps in the article. Worked fine. You may have skipped the step where it has you attach to one of the image processes.

The issue is that the coarray images aren't in the thread that the debugger starts in - they're actually in separate processes. This is why just setting a breakpoint in the initial thread doesn't work.

[jimdempseyatthecove]

Look at https://docs.microsoft.com/en-us/visualstudio/debugger/debug-multiple-processes?view=vs-2015#BKMK_Automatically_start_an_process_in_the_debugger

Jim Dempsey

[Jon_D]

Thank you Steve and Jim.

Steve, based on your suggestions, I was able to get the debugging working on a simple example. However, when I try it on my actual application which is quite large I get an error saying that the process can't be attached because there is not enough storage. This is with the 32bit compilation.

I thought that maybe this will be resolved with the 64bit compilation. However, when I try it with the 64bit compilation, nothing happens. It appears as if the program hangs. Even the simple example doesn't work (with or without your debugging code)  when I compile the code as 64bit application. Do you know why this might be happening? I have multiple versions of the compiler installed (both v17 and v19) with VS 2017 Community Edition.

Thanks,

Jon

[Steve_Lionel]

I did my test as 64-bit. I can't explain the "not enough storage" issue. For the test program I used the mcpi_coarray_final.f90 from the coarray_samples folder in the current samples bundle.

[Jon_D]

I am only getting a black screen with the following program when compiled as 64-bit. No errors, on messages, nothing. I uninstalled all versions of the compiler and reinstalled 17.0.8.275 (I need to use this version because a bug I reported hasn't been fixed yet with the newer compilers). Project file is also attached. Is there anything I need to do with the VS flags to get the 64bit run?

PROGRAM Test 
    WRITE (*,*) 'I am image ',THIS_IMAGE()
END 

[Steve_Lionel]

The project looks fine. What happens if you try to build and run this from the command line? (Use /Qcoarray). I suspect an issue with Intel MPI. Do you have any non-Intel MPIs installed?

[Jon_D]

Same problem when built from the command line. 32bit runs fine, 64bit hangs. With 64bit, I can see all the images in the Task Manager but command window shows nothing. 

I myself never installed another mpi knowingly, but looking at the Path variable, I see MS MPI installed. When I first started playing with coarrays, I was having an error (something like "incorrect parameter - localhost") even with the 32bit but was able to get rid of it by changing the order of the listed MPIs in the path as described in another forum discussion. 

[Steve_Lionel]

It sounds as if you're running into the problem noted in https://software.intel.com/en-us/forums/intel-visual-fortran-compiler-for-windows/topic/703331 See if the workaround in post 7 there works for you. 17.0.8 shouldn't have this problem .

[Jon_D]

Steve,

The workaround (setting I_MPI_COLL_INTRANODE=pt2pt) worked at the command line but I wasn't able to figure out how to set this variable in Visual Studio.

Assuming the issue is with the mpi library that came with compiler v17, I uninstalled it and installed 19.0.1. Now, when I compile and run my simple test in VS, 32bit prints incorrect image numbers (all 1's), 64bit still doesn't work. When I compile the two from the command line, 32bit runs but still with incorrect image numbers printed; 64bit runs correctly (without having to set I_MPI_COLL_INTRANODE) but produces a warning ("I_MPI_SCALABLE_OPTIMIZATION environment variable is not supported").

So, I am not sure what to do at this point. I can't tell what is broken: the specific distribution of the mpi library, integration with VS, some software-hardware incompatibility, or something else?

Thanks for any suggestions in advance.

Jon

[Steve_Lionel]

The warning is new in 19.0.1 and can be ignored for now.

To set the environment variable in VS. go to the Debugging property of the project and set Environment to I_MPI_COLL_INTRANODE=pt2pt

The printing of all 1s as image numbers sounds like the same issue as in https://software.intel.com/en-us/forums/intel-visual-fortran-compiler-for-windows/topic/801082 No resolution, though. My guess is that some other MPI is being seen first. Check the value of PATH and remove any MPI run-time that isn't the latest Intel one.

[Jon_D]

Steve,

You are right... That forum topic is describing the exact same problem I am having. It is unfortunate that it hasn't had any resolution. If Lorri wants me to make additional tests as a follow up to that topic, I am willing to do them.

I checked and double-checked my Path variable. The only other MPI I see is from Microsoft. I removed it from Path but it didn't work. I even renamed the MS mpi executables, but that didn't work either. The only thing that works is to set I_MPI_COLL_INTRANODE=pt2pt at the command prompt (setting it in VS as you suggested did not work). 

[Jon_D]

BTW, I reverted back to compiler version 17.0.8 and printing of 1s are resolved for the 32bit. I only have to figure out why 64bit hangs. Thanks!

[jimdempseyatthecove]

>> I only have to figure out why 64bit hangs.

I had an issue where an MPI function/subroutine required an integer argument of size equal to that of a pointer/ IOW for 64-bit build a 64-bit integer, however the call was made using a 32-bit integer. For some reason the interface checking did not expose the incorrect argument issue. Likely due to incomplete C interoperability argument checking.

Jim Dempsey

[Steve_Lionel]

Regarding the environment variable, I guess setting it in VS applies only to the process for the starter thread. What you can do instead (I think) is set it in the command environment and then run msdev to start VS. I think it will apply then automatically.

The 32-bit environment uses an old Intel MPI runtime, since Intel MPI dropped 32-bit support a few years ago. This is why you might see a difference.

I recommend opening an issue at the Intel Online Service Center to make sure it gets looked at. If you get a resolution, please report back here!

[Jon_D]

Hi,

It's been a while but I wanted give an update on my issues with coarrays. 

The Intel Fortran compiler I have been using is version 2017 Update 8. It turned out that the problems I was having were due to the number of images. Another forum topic stated that more than 16 images created issues. When I set the number of images to 10 in my coarray application (by default, I get 24 images on my computer), all the problems I was having went away and I was able to compile and run my program with no problems, both 32bit and 64bit versions.

Now, I am trying to migrate my program to Fortran 2019 Update 3. It appears that the limit on the number of images has been resolved. However, when I run the program, I get an MPI warning that states "I_MPI_SCALABLE_OPTIMIZATION environment variable is not supported.". Otherwise, the program runs without an issue. It would be nice, though, not to get this warning since I will be shipping this program to other people and I am sure I will get questions from them. This is with the 64bit version of the program. 32bit version of the program crashes. It appears there is a major issue with 32bit version of the coarrays. When I try a simple program which prints "I am image ...", I get "I am image 1" for 10 times.

Thanks,

Jon 

[Steve_Lionel]

The warning can be ignored for now, it will be fixed in a future update. I thought I had seen somewhere an environment variable could be set to suppress that, but I can't find it.

Likewise the 32-bit issue should be resolved in the next update. It's frustrating, I know. File an issue with Intel support to get more feedback.

[Jon_D]

Thank you, Steve.

There is one more thing that you may be able to shed some light on: With ifort 2017 Update 8, when I ran the program with 10 images, there would actually be 11 images generated. In a Resource Monitor showing all the processes running on the computer, 11th image would show as grayed out. With ifort 2019 Update 3, there are still 11 images but none of them are grayed out. After a few minutes into the run, one of the images automatically gets terminated. When the program is run from a batch file, this triggers execution of the proceeding commands within the batch file prematurely. This is an issue because those proceeding commands are supposed to process the output from the coarray program, which hasn't finished running yet.

Do you think this is another bug, or can it be somehow fixed within my Fortran code or setting some flags in Visual Studio.

Jon