Intel® Fortran Compiler
Build applications that can scale for the future with optimized code designed for Intel® Xeon® and compatible processors.
28384 Discussions

Compiling with OpenMP directives : debugger provides wrong line numbers

SCHWOERTZIG_T_
Beginner
604 Views

Hi,

I have some difficulties due to OMP directives.  I am compiling on linux64 with ifort 12.1.3.293 build 20120212 and following flags:
 -g -O0 -fp-model precise -openmp  -Dmkl -ftz -extend-source -DFCCI -check uninit -fpe0 -ftrapuv  -traceback

. I am stepping into a subroutine during debug (idbc or gdb). But both debuggers provides exactly the same line number (888) which does not correspond to the called subroutine. The called subroutine is written in a source file which contains 5 subroutines. I am calling the 5th one which start at line 1567. Line 888 is the middle of the 3rd subroutine.

Each subroutine in this source file uses OMP directives :
!$OMP DO SCHEDULE(DYNAMIC,1)
[fortran source]
!$OMP END DO

The problem disappear if I remove these OMP directives. Line number is also consistent with source file.
Is it expected or a known issue ?

Thanks for any help.
Thierry

 

0 Kudos
14 Replies
Lorri_M_Intel
Employee
604 Views

It's possible.  We have had some issues in the past with the line number correlation being incorrect when debugging.

I don't see where we've fixed any of those since 12.1, so updating to a new compiler is not likely to help.

I appreciate that you can't share the full project, but can you characterize the program better?  Is line 888 legitimately somewhere in the backtrace?   If not, is that third subroutine (where line 888 lives) also called in the same OMP region as the fifth subroutine?  Any other clues? 

In general, the previous problems were caused by a "stale" line number being reused, and that's what I'd expect was happening here too.

                   Thanks --

                       --Lorri

0 Kudos
SCHWOERTZIG_T_
Beginner
604 Views

Hi Lorri,

Thank you for this feedback.

The situation occurs with 1 threads. Subroutine_5 is called first. but i step directly on line 888.

Furthermore even if i create the breakpoint before running the program with idbc, the line number is not correct (888 again) :

[schwoertzig]$ idbc --args ./my_program -nthread 1
Intel(R) Debugger for applications running on Intel(R) 64, Version 12.1, Build [77.329.14]
------------------
object file name: ...
 Reading symbols from ...done.
(idb) break SUBROUTINE5
Breakpoint 1 at 0x656f38: source.F, line 888.
(idb) bt
The "backtrace" command has failed because there is no running program.

Here are line details from source.F :
Subroutine 1 : line 1-361
Subroutine 2 : line 368-722
Subroutine 3 : line 729-1226
Subroutine 4 : line 1233-1560
Subroutine 5 : line 1567-1688

Here is an extract of the flowchart.

 |   +--SUB_A
 |    |   +--SUB_A
 |    |    |   +--Subroutine 5
 |    |    |   +--Subroutine 4
 |    |   +--SUB_C
 |    |    |   +--Subroutine 3
 |    |   +--Subroutine 1
 |    |   +--Subroutine 2

Do you need more information ?

Thank you and Best Regards
Thierry

 

 

 

0 Kudos
SCHWOERTZIG_T_
Beginner
604 Views

Moving Subroutine5 to a new source file also fixes the issue. Debuggers are able to locate the correct line number corresponding to the subroutine beginning.
Does anybody have an explanation taking into account this new clue ?

Thanks and Best Regards
Thierry

0 Kudos
TimP
Honored Contributor III
604 Views

Are you telling us in-lining affects your debugging?

How about

-debug inline-debug-info

or 

-fno-inline-functions

if you don't want to compile procedures separately?

0 Kudos
SCHWOERTZIG_T_
Beginner
604 Views

Tim Prince wrote:

How about
-debug inline-debug-info

Thank you for this idea.
I just tested thses compilation options without success :
  -Ob0
  -debug inline_debug_info
  -inline none
  -inline-debug-info

  • Idbc –args ./myprogram –nthread 1

(idbc) break SUBROUTINE_5

Breakpoint 1 at 0x5f6b08: source.F, line 888

This is still the unexpected line number located in the middle of SUBROUTINE3 (in the same source file).

Thierry

0 Kudos
jimdempseyatthecove
Honored Contributor III
604 Views

Thierry,

Are these subroutines in a module? If so, and I've experienced this myself (and I imagine others here have too), it is possible that the output directory for your .mod and .o files is not the input folder for the link phase of the .o files. In this situation you end up compiling the new code and linking the old code. A quick test for this is to insert a "write(*,*) 'new code'" somewhere in the new code that gets executed early. If this write does not occur, then check your paths as this would indicate you are linking old object files.

Inlining and any optimizations will affect line numbering. A technique I often use is to insert a CALL to a subroutine that is explicitly stated as to not be inlined (and additionally not visible to IPO as it aggressively inlines). The called routine doesn't need to do anything. Usually, the line number where the call is made is always available. Once at break point, you can then single step (and disregard information presented as to line number).

Jim Dempsey

0 Kudos
SCHWOERTZIG_T_
Beginner
604 Views

jimdempseyatthecove wrote:

Thierry,
Are these subroutines in a module?
...
Jim Dempsey

Hi Jim,
Thank you for this information.
In my case subroutines are not in a module. I also compiled from scratch when i tested the inlining flags.


Using your tick, I added CALL VOID() (on line 1615) at the beginning of the SUBROUTINE5 (which beginns on line 1567).  I made a breakpoint inside VOID SUBROUTINE which is empty. Using "up" command to move in the backtrace debugger says to be on line 935 which is again somewhere in the middle of SUBROUTINE3 which is not related to SUBROUTINE5. This is really enigmatic.

Thierry

 

0 Kudos
jimdempseyatthecove
Honored Contributor III
604 Views

Thierry,

Looks like compiler optimization issue (rearranging code but not line numbers).

One other potential issue is: does the statement following the CALL VOID() contain a "variable" than is a statement function?

Some older programs made use of statement functions. These appear in the data declaration section and when immediately in front of code, are often confused with code.

AVG(A,B,C) = (A+B+C)/3.

The above, when appearing in the data section is a statement function (A,B,C must have been declared).
If the above appears after the first code statement, then it is a statement (not a "statement function").

When debugging, the debugger might hop to the location of the declaration of the statement function (then hop back on next step).

Jim Dempsey

0 Kudos
SCHWOERTZIG_T_
Beginner
604 Views

Hi Jim,

Thank you for the idea. There are only Data statments before and no such situation after that. (Replacing data statments does not change the behavior.)

I found a second workaround :
In SUBROUTINE_2 there are 2 !$OMP DO block :

!$OMP DO SCHEDULE(DYNAMIC,1)
      DO IG = 1, NG
      ...
      END DO
!$OMP END DO
C
!$OMP DO SCHEDULE(DYNAMIC,1)
      DO IG = 1, NG
      ...
      END DO
!$OMP END DO

Removing $OMP END DO directives also fixes the issue : Breaking into SUBROUTINE_5 leads to the correct line number (1567 instead of 888).

Thierry

 

0 Kudos
jimdempseyatthecove
Honored Contributor III
604 Views

That should be sufficient information for Intel support to go on, but they are hesitant to do anything without a simple reproducer. If you can put together a simple reproducer then they will likely fix the problem.

Jim Dempsey

0 Kudos
Kevin_D_Intel
Employee
604 Views

I created a much shorter test program based on all the details in this thread but was unable to reproduce the failure so my test is not imitating enough of the original code to produce the line number confusion. IDB is finds/sets the break at the correct line number for subroutine 5 in my case with 14.x and older 12.1 compilers.

Is there any chance at getting a reproducer?

You can provide those in private via a private message or submitting an issue under Intel Premier Support.

0 Kudos
Lorri_M_Intel
Employee
604 Views

Thank you Jim - couldn't have said it better myself!  :-D

Yes, Thierry, I tried to create my own reproducer given the patterns you've described, with no luck.
I don't need a runnable reproducer, just something I can compile and look at our internal tool that "tracks" line number<->source correlations.

And, you can use the "Private Message" utility to send it along for further security.

           Thank you --

                                     -- Lorri

0 Kudos
SCHWOERTZIG_T_
Beginner
604 Views

Hi,

I built a small example using OMP directives. This source code structure is a litle bit different from the original description but the consequences are similar ones. Debugger has some trouble to match source lines.

I am not sure if I did something wrong or if it is highlithing a compiler bug. You will find below all needed information.

           --- Thanks and Best regards.

                          --- Thierry

SOURCE FILE : main.F

      PROGRAM TOTO
      CALL SIGEPS51()
      END PROGRAM

SOURCE FILE : sigeps51.F

      SUBROUTINE SIGEPS51()
      IMPLICIT NONE
      INTEGER I,J
      REAL*8 :: TFEXT, TFEXTT
      TFEXT  = 0.0D00
      TFEXTT = 0.0D00      
      J = 2
      I = 0
      IF (J==1)THEN
#include "atomic.inc"
          TFEXT = TFEXT + TFEXTT
#include "atomend.inc"
      ELSEIF(J==2)THEN
        DO I=1,2
#include "atomic.inc"
          TFEXT = TFEXT + TFEXTT
#include "atomend.inc"
        ENDDO
      ENDIF
#include "atomic.inc"
          TFEXT = TFEXT + TFEXTT
#include "atomend.inc"
      print *, "1-UN"
      print *, "2-DEUX"
      print *, "3-TROIS"            
#include "atomic.inc"
          TFEXT = TFEXT + TFEXTT
#include "atomend.inc"
      RETURN
      END SUBROUTINE

SOURCE FILE : atomic.inc

#if defined(CPP_mach)
         PRINT *, "CPP_mach=", CPP_mach
#else
!         PRINT *, "CPP_mach NOT DEFINED"          
#endif

#if defined(_OPENMP)
         PRINT *, "OPENMP=", _OPENMP
#else
!         PRINT *, "OPENMP - NOT DEFINED"          
#endif
!$OMP CRITICAL
!$OMP ATOMIC

SOURCE FILE : atomend.inc

!$OMP END ATOMIC
!$OMP END CRITICAL



My Compiling command is the following one :

[schwoertzig]$ ifort -debug inline-debug-info -g -O0 -fp-model precise -openmp  -Dmkl -ftz -extend-source -DFCCI -check uninit -fpe0 -ftrapuv  -traceback -c main.F;ifort -g -O0 -fp-model precise -openmp  -Dmkl -ftz -extend-source -DFCCI -check uninit -fpe0 -ftrapuv  -traceback -debug inline-debug-info -c sigeps51.F;ifort -g -O0 -fp-model precise -openmp  -Dmkl -ftz -extend-source -DFCCI -check uninit -fpe0 -ftrapuv  -traceback -debug inline-debug-info sigeps51.o main.o -o main
[schwoertzig]$ ifort -v
ifort version 12.1.3

 

The debugger behavior is the following one :

[schwoertzig]$ idbc ./main
breIntel(R) Debugger for applications running on Intel(R) 64, Version 12.1, Build [77.329.14]
------------------
object file name: ./main
aReading symbols from /home/schwoertzig/tmp/tests_fortran/bug_OMP/main...done.
(idb) break sigeps51.F:11
Breakpoint 1 (sigeps51.F:11) pending
(idb) break sigeps51  
Assertion failed: "!"should not be possible"" ./src/st/mem/semanticevents.C:435
This is an unexpected condition and may indicate the presence of a defect.
If you wish to report this, please include the stack trace that follows.
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN15IDBAssertFailed3runEPKcS1_j+0xe) [0x60dca2]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN14DTLU_namespace12assertFailedEPKcS1_j+0x1e) [0xb51f8e]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb() [0x8e3e3d]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_Z17nextSemanticEventP14ReadableSymtab13RelocatedAddr+0x2c9) [0x8e4b6f]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN38PositionWithRespectToProlog_Calculator9calculateEP9AddrSpaceRK13RelocatedAddrbmbmRP14SymbolEntrydefR27PositionWithRespectToPrologRS2_SA_SA_SA_+0x80f) [0x8f9d03]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN22ProcedureDescriptorDef27positionWithRespectToPrologEP9AddrSpaceRK13RelocatedAddrRP14SymbolEntrydefR27PositionWithRespectToPrologRS2_SA_SA_SA_+0xa5) [0x8fa507]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN14IfOccursInExpr9getRAListEP6VectorI13RelocatedAddrE+0x106f) [0x648a53]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN21AbstractConditionExpr9getRAListEP6VectorI13RelocatedAddrE+0x2a) [0x9433c2]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN19AbstractConditionPC7realizeEv+0x9d) [0x942ad9]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN20AbstractHandlerTable6createElbP17AbstractConditionP3CmdbbN7Handler15HandlerDurationES3_bP9ConditionNS4_18UserConditionStyleElNS4_14PassCountStyleE+0x473) [0x94a1cd]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN11CmdGdbBreak5do_itER19CmdExecutionContextRN10BaseForCmd9CmdResultE+0x274) [0x64e432]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_ZN10BaseForCmd7executeEb+0xfc5) [0x6598e5]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb() [0x60b74c]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_Z15ProcessCommandsv+0x54) [0x60ba5c]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(__gxx_personality_v0+0x41d) [0x6099e5]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(_Z7idbMainiPPKcS1_+0x180) [0x609c76]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(main+0x3a) [0x60998a]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d2de1ed1d]
/opt/intel/composer_xe_2011_sp1.9.293/bin/intel64/iidb(__gxx_personality_v0+0x232) [0x6097fa]

 

0 Kudos
SCHWOERTZIG_T_
Beginner
604 Views

Here is the simpliest reproducer I found. For debuggers (idbc/gdb) line 6 does not exist. It should be "A = A + B". But Replacing include statments by the corresponding directive seems to work. How to get rid of this issue ? Does anyone know a workaround ?

     Thanks

            Thierry

      PROGRAM ATOMIC
      INTEGER :: A, B
      A  = 0
      B  = 0     
#include "atomic.inc"  !contains only single line !$OMP ATOMIC
          A = A + B
#include "atomend.inc" !contains only single line !$OMP END ATOMIC 
      END PROGRAM

[schwoertzig]$ ifort -debug inline-debug-info -g -O0 -openmp  main.F
[schwoertzig]$ idbc ./a.out
Intel(R) Debugger for applications running on Intel(R) 64, Version 12.1, Build [77.329.14]
------------------
object file name: ./a.out
Reading symbols from /home/schwoertzig/tmp/tests_fortran/bug_OMP/source_code/a.out...done.
(idb) break main.F:6
Breakpoint 1 (main.F:6) pending

 

0 Kudos
Reply