Debugging Fortran applications on Intel platforms running Linux

[Intel_C_Intel]

Hi, I'm Gordon, the Debugger Consultant for Intel's Software Products Division. I'm responsible for enabling and promoting effective debugging on Intel platforms in general, and with the Intel Debugger in particular.

My current focus is on creating a repository of Best Known Methods for debugging serial, thread parallel and cluster parallel applications.

I'd like to get a feel for how the community at large approaches application debugging. I'm curious to know things like:

• What types of bugs do you encounter most often?
• What types of bugs do you find most difficult to locate and fix?
• What methods do you use to accomplish this?
• Which debuggers do you tend to use (and why)?
• Which debuggers do you prefer to use (and why)?
• Which debugger features do you find yourself using most often?
• Which new debugger features would you consider to be essential?

Please feel free to contribute accordingly. I appreciate your participation, it helps significantly to improve both the processes and the tools for all of us.

Thanks!

[mmquan]

I am new here. Now I am doing a project with both Fortran and Java. My response is rewrite a Fortran program with Java programming language. It is not difficult to rewrite the source code, but when I finished the main program I found some problems. I can't get exactly the same output in Fortran and Java, actually very big difference. During the course, I have to debug. Actually I didn't use any debug tools, but I think I need a debugger. I don't know now whether it's the problem of Intel Fortran Compiler(ver. 7.1) as I said on the other message in the forum. But there is quite a lot of variables and arrays in the fortran program waiting for me check whether they get different in both Fortran and Java versions. Usually what I did is output all the variables and arrays to a file and then use diff in Linux to compare, and that means I have to write this function by myself. Some times there are too many difference between the two versions then I feel quite tedium. Actually I am a starter of programming, and up to now I still don't know the importance of a debugger, but I will say if I have a debugger which can give me a choice to output some or all the variables to a binary file with a breakpoint, then I guess I will like it very much! Maybe there is already such a tools, but I just said what I feel like.

[mbkennel]

I actually find it rare to use the debugger. I almost never use the Intel debugger, though I haven't ever made multiprocessing codes yet.

It would be nice if the executable could be debugged if it were in a different directory from where it was built. My build-system compiles a number of executables and moves them into a 'bin' directory.

Why? Because I then run the programs (often as they are in a path) on data files with long command lines with various options.

That brings up another wish. I have shell scripts et al running the calculations---they often insert a number of command lines options. I would love it if I could change
"myproram -blah blah blah blah" to "intel_debug myprogram blah blah blah'

and have everything work. I.e. have it be just as easy as /bin/time



What I really need is extremely reliable and precise compiled-in checking (as in -C). Unfortunately not all -C options are working yet.

[Intel_C_Intel]

Mmquan,

The debugger isnt (and shouldnt be) the only debugging tool in a programmers toolbox. But it is useful in many cases. For example, in the scenario you mentioned, the Intel Debugger gives you the ability to specify things like:

when quiet at 200 { printf "At line 200 the value of x is 0x%x, of i is %d ", x, i }

In this way you can selectively output the variables of interest at the points of interest. If you do basically the same in whichever debugger youre using for Java, then youll have something of interest to compare.

Mbkennel,

The Intel Debugger has three mechanisms for specifying the location of corresponding source files. The -I option, the use command and the map source directory command (documented here, here and here). These features are designed to enable you to do exactly what youre describing.

In regard to modifying your script to run your program under debugger control, I cant currently offer you exactly what youre looking for (although that would be nice Ill submit a corresponding feature request). However, I can come close in two different ways.

1. You can say:

idb program-name <<$EOD
run arg1 arg2 arg3
$EOD

2. You can also create a .dbxinit initialization file in the directory from which you run the program. In that file, specify the run command immediately followed by the arguments to your program (on the same line as above). Then, when you do idb program-name, the debugger will automatically run the program with the specified arguments and prompt you when its done (or when it encounters a problem).

There are probably other ways as well, but hopefully this will get you started (and keep you moving).

Peace,

-- G

[Intel_C_Intel]

Mmquan,

The debugger isnt (and shouldnt be) the only debugging tool in a programmers toolbox. But it is useful in many cases. For example, in the scenario you mentioned, the Intel Debugger gives you the ability to specify things like:

when quiet at 200 { printf "At line 200 the value of x is 0x%x, of i is %d ", x, i }

In this way you can selectively output the variables of interest at the points of interest. If you do basically the same in whichever debugger youre using for Java, then youll have something of interest to compare.

Mbkennel,

The Intel Debugger has three mechanisms for specifying the location of corresponding source files. The -I option, the use command and the map source directory command (documented here, here and here). These features are designed to enable you to do exactly what youre describing.

In regard to modifying your script to run your program under debugger control, I cant currently offer you exactly what youre looking for (although that would be nice Ill submit a corresponding feature request). However, I can come close in two different ways.

1. You can say:

idb program-name <<$EOD
run arg1 arg2 arg3
$EOD

2. You can also create a .dbxinit initialization file in the directory from which you run the program. In that file, specify the run command immediately followed by the arguments to your program (on the same line as above). Then, when you do idb program-name, the debugger will automatically run the program with the specified arguments and prompt you when its done (or when it encounters a problem).

There are probably other ways as well, but hopefully this will get you started (and keep you moving).

Peace,

-- G

[ryofurue]

Hi Gordon,

I use debuggers mostly when I encounter signals (segmentation fault, etc.) and floating point exceptions (mainly overflow and invalid operation). A debugger is nice because it tells you which line of your source code causes the signal or exception.

So, it would be nice if you can change the IEEE floating point mode from within the debugger. The default IEEE mode
is to continue even when overflow or invalid operation occurs.

Oh, and I don't like a debugger which requires source files. I used to use gcc (g++) and gdb (GNU debugger). With gcc, you compile your source code with the -g option, and then the source code is embedded in the executable. So, you don't need to carry around the source files or to tell the debugger their locations. This is important because my source files are scattered (I use my own little libraries).

By the way, I envied my wife, who were using Visual Basic. Not that I liked the language, but that the debugger was fabulous. When an error occurs, the line of the source code is highlighted, and when you bring your mouse pointer on a variable name, its value is shown in a baloon! Compared to that, what I'm doing (use the write statement to print the value of variables to debug) is by far more inefficient.

I haven't touched the Intel debugger, but what is it or will it be such a graphical debugger?

Ryo

[Intel_C_Intel]

Furue,

Thanks for the suggestion regarding IEEE mode. I'll make a note for the engineering team.

As far as embedding sources in the executable goes, few of the commercial compilers (that I know of) will do this. In fact, the trend in the industry seems to be moving toward separating out from the executable anything that isn't absolutely essential to its normal mode of operation (including debug information).

This has some advantages, like being able to produce a fully optimized (production) version of an application with separate debug information, so that you can debug exactly what you'll be shipping (or otherwise deploying). In addition, you can then debug customer reported problems using that same (external) debug information.

Most Debuggers (including the Intel Debugger) allow you to specify the location of the sources in those cases where they're no longer maintained with the executable. This helps to solve the issue you sited. The Intel Debugger actually provides 3 such mechanisms for doing this (referenced in an earlier reply to this thread for your convenience).

As far as graphical interfaces go, the Intel Debugger does have a rudimentary built-in GUI, and it can be used from emacs, DDD and Streamline DDT.

In addition, in place of using "printf", the Intel Debugger allows you to "instrument" your code at run-time with "printf"-like statements that you don't have to compile in and out of you program (please see the related note in an earlier reply to this thread).

Please give us a try and let me know what think.

Thanks.

-- Gordon

[Intel_C_Intel]

Furue,

Thanks for the suggestion regarding IEEE mode. I'll make a note for the engineering team.

As far as embedding sources in the executable goes, few of the commercial compilers (that I know of) will do this. In fact, the trend in the industry seems to be moving toward separating out from the executable anything that isn't absolutely essential to its normal mode of operation (including debug information).

This has some advantages, like being able to produce a fully optimized (production) version of an application with separate debug information, so that you can debug exactly what you'll be shipping (or otherwise deploying). In addition, you can then debug customer reported problems using that same (external) debug information.

Most Debuggers (including the Intel Debugger) allow you to specify the location of the sources in those cases where they're no longer maintained with the executable. This helps to solve the issue you sited. The Intel Debugger actually provides 3 such mechanisms for doing this (referenced in an earlier reply to this thread for your convenience).

As far as graphical interfaces go, the Intel Debugger does have a rudimentary built-in GUI, and it can be used from emacs, DDD and Streamline DDT.

In addition, in place of using "printf", the Intel Debugger allows you to "instrument" your code at run-time with "printf"-like statements that you don't have to compile in and out of you program (please see the related note in an earlier reply to this thread).

Please give us a try and let me know what you think.

Thanks.

-- Gordon

[Intel_C_Intel]

Furue,

Thanks for the suggestion regarding IEEE mode. I'll make a note for the engineering team.

As far as embedding sources in the executable goes, few of the commercial compilers (that I know of) will do this. In fact, the trend in the industry seems to be moving toward separating out from the executable anything that isn't absolutely essential to its normal mode of operation (including debug information).

This has some advantages, like being able to produce a fully optimized (production) version of an application with separate debug information, so that you can debug exactly what you'll be shipping (or otherwise deploying). In addition, you can then debug customer reported problems using that same (external) debug information.

Most Debuggers (including the Intel Debugger) allow you to specify the location of the sources in those cases where they're no longer maintained with the executable. This helps to solve the issue you sited. The Intel Debugger actually provides 3 such mechanisms for doing this (referenced in an earlier reply to this thread for your convenience).

As far as graphical interfaces go, the Intel Debugger does have a rudimentary built-in GUI, and it can be used from emacs, DDD and Streamline DDT.

In addition, in place of using "printf", the Intel Debugger allows you to "instrument" your code at run-time with "printf"-like statements that you don't have to compile in and out of your program (please see the related note in an earlier reply to this thread).

Please give us a try and let me know what you think.

Thanks.

-- Gordon

[mbkennel]

specifying the location of the sources with a debugger option works OK if it is a single executable program.

However, especially with numerical computations, the executable is called in the middle of a sequence of commands orchestrated by a shell or perl script, and often have many command line options for the program.

As i said before it would be very nice to be able to very simply add a "idb -- " or something before the normal command line in the script (minimal intervention) and have it Just Work.

maybe -ginclude-source as an additional option, to keep the source code along with the executable would be nice?

Also in numerical computation it may be frequent for the executable to run on different machines than the build machines (e.g. cluster situations) and require debugging there and not on its original build environment. Again, it's much more convenient if the source were automatically included.

[martin_]

Hi Gordon,

I think it is important to have a debugger which not only debugs but also helps optimize your code. My main experience with debugging/optimizing multithread openmp code has been on an Origin series machine from SGI (so no intel compilers!). Here I used SpeedShop which for each thread produces time sampled output (you can set it to find cache misses, page faults etc.). Analysis of this output allows you to track not only where in your code any bottlenecks are but what is causing the bottleneck (for my code generally having to reload from cache). You can then alter your code if necessary to improve performance. I would strongly encourage any debugging software Intel develop to have such an ability (if you don't have it already). For most applications debugging is not sufficient but optimizing is. Indeed I have two books for parallel programming. One on OpenMP and one three times as large on how to optimize code.

Martin.

[Intel_C_Intel]

MB,

Regarding the location of the sources, the debug information in the executable does contain a reference to the original location of the sources (either absolute or relative depending on compiler options). In addition, the Intel Debugger "use" list and "map source directory" list both act as true PATH style search lists and apply to any and all processes in a multi-process application. This will work exactly as you need it to (including the clustered application case).

However, I'll submit a feature request on your behalf thatsomething like -ginclude-source be considered as an option for a future release.

Regarding the ability to simply precede your command line with "idb" to invoke the debugger, please comment on how the work-arounds I previously suggested fall short of your goals. Do they not work for you at all? Do they work but prove to be too inconvenient to use on a regular basis? We'd like to know so that we can improve on this for you.

Thanks.

-- Gordon

---

Martin,

Intel produces several Software Development Products that help you to tune your application to run optimally on Intel platforms. Just to name a few, there are:

• The Intel Compilers that provide High Level Optimization assistance with reports, command line options, pragmas, Inter-procedural Optimizations, Profile Guided Optimizations and more (including OpenMP)
• High performance libraries for primitive and mathematical operations
• VTune Performance Analyzer
• Intel Threading Tools: Thread Checker and Thread Profiler
• Intel Cluster Tools: Trace Collector and Trace Analyzer

I believe that you'll find references to a wealth of such tools and related support services on the Intel Software Development Products web pages.

Please let me know if you have any specific questions about any of these tools.

Thanks.

-- Gordon

[Dharhas_P_]

Hi,

First of all to answer your questions in the past I have mainly just used a text editor and print statements to debug code but now I am trying to transition to using gdb or idb to make things more efficient. There are still places where outputting a whole bunch of stuff to a text file and reviewing it later makes sense. In the past some of the most difficult to trace errors occured in a mixed fortran/c++ environment when variables didn't transfer correctly between the fortran and C++ code.

Presently I am very adept at working with Emacs and using print statements to debug but I am looking into options to find a GUI IDE that supports Fortran 77/90 and runs on Linux. Any suggestions would be welcome.

I do have a major issue right now that I could use some help with. I have a numerical modeling code that is segfaulting on my system after 2 days or so of running. Is there anyway to use idb to see what is happening at that point and where it is crashing. I'm not sure how to set a breakpoint for an issue like this. This has been a very stubborn bug that I now think is related to disk IO since the same simulation runs fine on machines at another location and here it runs fine if I disable all but one of the outputs.

thanks,

- dharhas

[Intel_C_Intel]

Hi dharhas,

Thanks for participating.

Both gdb and idb work from within emacs. They also both work from within ddd. So you do have some options here. But if your mixed language debugging involves f90 application code, idb is probably your better bet.

Using debugger commands like:

when quiet at 200 { printf "At line 200 the value of x is 0x%x, of i is %d ", x, i }

Is a useful alternative to (and more flexible than) hardcoded printf statements. In fact, IDB can even be configured to log that output to a file for later examination. You could even emit progress markers to make later analysis more effective.

The long running application is an interesting case. I'd start by trying to determine if the disk that you're eliminating from the run (to avoid the failure) produces a failure with a small test case that exhibits a similar usage pattern.

If that doesn't help, I'd try to determine (from the details of the output of the last crash) approximately where in the code it's crashing. Then you could rerun under debugger control with a breakpoint set a few lines before that. Even if you miss, once the application crashes, the debugger takes control, at which point you can examine a backtrace of the stack to see exactly how you got to that point.You can even examine the surrounding context (i.e. call parameters, stack local storage, heap storage and static storage) at each level in the call chain (working your way back from the point of failure) to determine the exact cause of the failure.

Please let me know what you find and if you need further suggestions.

Thanks.

-- Gordon