Intel® Moderncode for Parallel Architectures
Support for developing parallel programming applications on Intel® Architecture.

Breakpoint error (159) during MPI run

Anders_S_1
New Contributor III
10,049 Views

Hi,

I am running Fortran programs under Windows 7 parallelized with Intel MPI.

Program 1 runs OK with size set to 1 and up to  8, respectively (four cores 4790K).

Program 2 runs OK for size=1 but for size=2-8 I get the error

forrtl: severe (159): Program Exception breakpoint .

I am compiling and linking the program i Visual Studio 2013.

I am using version Visual Fortran 64-bit 17.0.4 Package ID: w_comp_lib_2017.4.210.

Best regards

Anders S

0 Kudos
1 Solution
jimdempseyatthecove
Honored Contributor III
10,007 Views

>>What is the MPI configuration file used for in the Compiler Fortran language?

https://software.intel.com/en-us/forums/debug-solutions/topic/673726

https://software.intel.com/en-us/node/610381

>> I launch the program with mpiexec in a bat file

If you do this, you also must assure the environment is setup correctly. One way to do this is via the Start menu

Start | All Programs | Intel Parallel Studio XE 2017 | Compiler and Performance Libraries | Intel MPI Library | Intel64 Build Environment

Then CD to your application folder and execute the .bat file.

*** Note,

>>forrtl: severe (159): Program Exception breakpoint

When starting from the .bat, the MS VS debugger is not running, but the process terminated due to an exception.

Also, note that when configuring MS VS to start your application with mpiexec or mpirun, the your application starts as a separate process. IOW the process being debugged is mpiexec, and not your application.

Your build options indicated you included /traceback. Did the console window show any traceback information?

Note, you can use Attach to Running Process with Visual Studio Debugger.

https://msdn.microsoft.com/en-us/library/3s68z0b3.aspx

The trick is to attach to the process prior to crash.

One way to do this is to add a Fortran function, name it something like WaitingToGo, that returns .true.
Add to PROGRAM:

  DO WHILE(WaitingToGo())
  ENDDO

Assure that the subroutine is in a separate file
Assure that IPO is disabled

Now then you can now use mpiexec to start the process, specify 1 rank (1 process)
The program will now hang in an infinite loop.

Use the MS VS Attach to Running Process (link above), issue Break all, issue single step until you reach the CALL to WaitingToGo (the DO WHILE loop), then use the debugger to set next statement to statement following the ENDDO then continue,

*** Note, if you start multiple processes, you can launch a 2nd, 3rd, .. MS VS instance and attach to each of the spawned processes (assure that you attach to different ID's of the named process).

Coordinating the "continues" could be problematic with more than a few processes. For this I would suggest performing the WaitingToGo only from rank 0, then immediately following that have an MPI synchronization point. You can attach to the additional processes (from additional MS VS IDEs) and not issue the break all (and reposition the next statement).

Now then, when you continue the rank 0 process all the others resume. Then when the exception occurs, hopefully it will show up in one of the debuggers (shortly afterword one or more other processes may experience an MPI timeout, which you ignore).

Jim Dempsey

View solution in original post

0 Kudos
27 Replies
jimdempseyatthecove
Honored Contributor III
7,441 Views

What were your build options?

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

Could you be more precise. What kind of info do you want me to send over? Just for the build step the configuration manager says Debug and x64. 

Best regards

Anders S

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

Please find attached the settings of debug, compiler and linker. In addition, I have joined the beta testing of 2018 Fortran compiler giving the possibility to find out if the breakpoint problem appears there too.

Best regards

Anders S

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

I get the same breakpoint problem with the 2018 beta compiler version.

Best regards

Anders S

0 Kudos
jimdempseyatthecove
Honored Contributor III
7,442 Views

>>Program 2 runs OK for size=1 but for size=2-8 I get the error
>>      forrtl: severe (159): Program Exception breakpoint .

size=

What do you mean by this? Number of ranks, or number of OpenMP threads?

How are you launching the program?

Command line, or from VS IDE?

Are you launching the program directly or via mpirun/mpiexec?

You should be aware that when you run a thread-parallel program (OpenMP here) that you normally link with the serial version of MKL. Conversely, when you run a serial program that you normally link with the parallel version of MKL. Running parallel and parallel can result in each OpenMP thread of your application launching an MPI instance with the number of hardware threads on your system (on your system 8x8=64 threads). Environment variable settings can vary this.

Note, if you are running from the IDE debugger .AND. launching the program via mpirun may be problematic as the program being debugged is mpirun and not the program(s) it launches in separate process(es).

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
7,441 Views

Hi Jim,

Let me clarify a little. I first parallellized the code with OpenMP and there is still some routines with OpenMP commands, therefore the OpenMP compile commands. The I went to MPI and made copies of the OpenMP subroutines and changed the MPI.

The exe-file is produced by Visual Studio and then I launch the program with mpiexec in a bat file. When I say size 2-8 I mean that I choose the size to be 2-8 (2-8 parallel processes).

I wrote earlier program1 and program 2 but there is only one program, that was run for two different cases. The breakpoint problem occurs in a subroutine that is only used in the second case  ("program 2).

I really appreciate working in the visual studio environment! Can I launch the program from VS? How do I do, if so?

What is the MPI configuration file used for in the Compiler Fortran language?

What do you recommend me to do?

Shall I take away all OpenMP subroutimes in the MPI-driven program?

Best regards

Anders S

0 Kudos
jimdempseyatthecove
Honored Contributor III
10,008 Views

>>What is the MPI configuration file used for in the Compiler Fortran language?

https://software.intel.com/en-us/forums/debug-solutions/topic/673726

https://software.intel.com/en-us/node/610381

>> I launch the program with mpiexec in a bat file

If you do this, you also must assure the environment is setup correctly. One way to do this is via the Start menu

Start | All Programs | Intel Parallel Studio XE 2017 | Compiler and Performance Libraries | Intel MPI Library | Intel64 Build Environment

Then CD to your application folder and execute the .bat file.

*** Note,

>>forrtl: severe (159): Program Exception breakpoint

When starting from the .bat, the MS VS debugger is not running, but the process terminated due to an exception.

Also, note that when configuring MS VS to start your application with mpiexec or mpirun, the your application starts as a separate process. IOW the process being debugged is mpiexec, and not your application.

Your build options indicated you included /traceback. Did the console window show any traceback information?

Note, you can use Attach to Running Process with Visual Studio Debugger.

https://msdn.microsoft.com/en-us/library/3s68z0b3.aspx

The trick is to attach to the process prior to crash.

One way to do this is to add a Fortran function, name it something like WaitingToGo, that returns .true.
Add to PROGRAM:

  DO WHILE(WaitingToGo())
  ENDDO

Assure that the subroutine is in a separate file
Assure that IPO is disabled

Now then you can now use mpiexec to start the process, specify 1 rank (1 process)
The program will now hang in an infinite loop.

Use the MS VS Attach to Running Process (link above), issue Break all, issue single step until you reach the CALL to WaitingToGo (the DO WHILE loop), then use the debugger to set next statement to statement following the ENDDO then continue,

*** Note, if you start multiple processes, you can launch a 2nd, 3rd, .. MS VS instance and attach to each of the spawned processes (assure that you attach to different ID's of the named process).

Coordinating the "continues" could be problematic with more than a few processes. For this I would suggest performing the WaitingToGo only from rank 0, then immediately following that have an MPI synchronization point. You can attach to the additional processes (from additional MS VS IDEs) and not issue the break all (and reposition the next statement).

Now then, when you continue the rank 0 process all the others resume. Then when the exception occurs, hopefully it will show up in one of the debuggers (shortly afterword one or more other processes may experience an MPI timeout, which you ignore).

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
7,441 Views

Hi Jim,

Yes, I got a normal traceback. The breakpoint was positioned at the last line (END) in a specific subroutine.

June 13 2017 I asked IDZ if VS could be used to produce an executable to be run by mpiexec and I got a positive answer from Tim P.. 

Is there any way to transfer the source file list in VS to a batch file? Then the method to run the program entirely from the command line should be more attractive,

Best regards

Anders S.

0 Kudos
jimdempseyatthecove
Honored Contributor III
7,442 Views

>>Is there any way to transfer the source file list in VS to a batch file?

I do not understand the question.

Do you mean the "run" command?

If so, insert into your .bat file the program name (qualify the path if bat file is run from different directory than executable)
Follow the program name with a space
Then from VS IDE, Right-Click on startup project (this may be your only project), select Properties, Configuration Properties, Debugging, and in there you will find your command arguments, select that, copy to paste buffer, then paste into the .bat file (in whatever editor you used to create the .bat file).

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

Thanks for your feedback on my questions!

You gave a link to how to configure Visual Studio with MPI, which I have not had the possibility to try yet due to a journey. Just to be sure, is this the way to be able to run MPI  from Visual Studio that I asked for?

Best regards

Anders S

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

I have isolated the part of the code giving rise to the breakpoint error and put it into a small subroutine PLLtest (attached). I defined a Quickwin project in VS2015, configured it with MPI (Cluster Studio 2017 update 4)(attached), added a driver (attached), created the PLLtest.exe in VS but run the executable from the command line. This gave two graphic windows with the correct text (START for rank=1, START+HELLO for rank=0) but the breakpoint error appeared (as expected).

Then I run the executable in VS by CTRL+F5 with WRITE(6,3) in the driver removed and got two windows with no text (attached screen view). When the WRITE(6,3) was active two graphic screen were obtained but only with the text START on both screens, no HELLO for rank=0.

My new question is now how to configure VS+MPI to get the correct text output when executing by Ctrl+F5.

The -localroot parameter was suggested by James in an earlier case. If localroot is removed no windows appear.

Best regards

Anders S

0 Kudos
jimdempseyatthecove
Honored Contributor III
7,442 Views

You have entered an area that I have not experienced (MPI QuickWin application launched in VS).

Observations:

When you launch the application from a command window via mpirun or mpiexec, the "run environment" consists of three windows: two QuickWin process windows, plus the command window that ran the mpirun/mpiexec program. In MS VS, when a project type is declared as being a Windows app (as opposed to console app), then no command window is instantiated, and thus has no place to write the console text to.

To correct for this (guessing here).

Add a new project, a console project, with a main and dummy PRINT "hello". This program is intended to NOT run. Set this project as your startup project... however, set the project Debugging Command property to launch your QuickWin program via mpirun/mpiexec. Note, you will have to path qualify the path to your QuickWin project and/or set the Debugging Working Directory property to that folder.

If that does not work,

Modify the Debugging Command property by inserting "cmd /K " on the left side ("cmd /K mpiexec ...")

The K is there to preserve the window.

Good luck

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

I have understood that not so many customers combine MPI and QuickWin. I do it because I can get graphics output during computation, which is a GREAT help to understand what is going on.

In an earlier case named "Can MPI be used to parallelize a QuickWin application" I was helped by James T to sort things out.

I am very eager to see the supplied program be run totally in the VS environment with the correct text output.

I will check your suggestions and respond.

Best regards

Anders S

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim el al,

If you do not disagree I will open a new case on running a Quickwin+MPI project in the VS or command line environment.

I tried your last idea but I am not sure I understood it.

Best regards

Anders S

0 Kudos
jimdempseyatthecove
Honored Contributor III
7,442 Views

The Windows program cmd.exe is a command prompt window (with command line in interpreter). This has also been miscalled a "DOS Box", and is what is used when you launch a Console Window application. PRINT *,"Hello" writes text to the application's console window (or redirected to file when console output redirected). A QuickWin program is a Windows application and does not have a Console Window, and thus the output goes to the NULL device. (IOW is consumed without error)

Therefore, if the next to the last suggestion does not instantiate a Console Window for mpirun (I think it will), then the last suggestion definitely will.

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

Thanks for your answer!

When I build and run the test program entirely in the command line environment everything works (text output and no breakpoint error)! 

Visual Studio environment:

The .exe file is built in VS.

a) Your suggestion with an extra console project:

I get one console and two graphic windows but only the text START appears, no HELLO text for rank=0. The program does not go to an end.

b) You suggestion with cmd /K:

I am not sure how to apply it.

I have attached the slightly modified test program now called main and screen outputs for the two VS cases.

Best regards

Anders S

0 Kudos
jimdempseyatthecove
Honored Contributor III
7,442 Views

From VS, on the

Configuration Properties | Command | cmd
Configuration Properties | Command Arguments | -K mpiexec.exe -localroot -n 2 C:\Data\... (you complete this)

Effectively you are inserting "cmd /K " in front of your equivalent comment line.

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
7,442 Views

Hi Jim,

I implemented your suggestion according to best understanding but the system does not accept the K parameter.

I supply the modified VS command , the response the the modified command and also the correct result when I run from the command line.

Best regards

Anders S

 

0 Kudos
jimdempseyatthecove
Honored Contributor III
7,442 Views

You did not follow instructions

Command is to be set to

     cmd

Command Arguments are to be set to

     -K mpiexec.exe -localroot -n 2 C:\Data\... (you complete this)

cmd.exe is a command line shell program. Typically run as

        cmd -cmdOption -cmdOption YourProgram yourProgramArg otherProgramArg ...

The cmd option -K, when used, must be the last cmd option

In your case, the "YourProgram" is mpiexec, and not you MPI program. Your MPI program is an argument to mpiexec

Jim Dempsey

0 Kudos
Anders_S_1
New Contributor III
6,656 Views

Hi Jim,

Sorry for not following instructions but I tried a number of interpretations of your suggestion.

Now I have made a new try according to you instructions but VS is not cooperating (see VS commands and VS respons attached)

Best regards

Anders S

0 Kudos
Reply