gdb-ia crashes when reading symbols from executable (linux)

[Jacob_L_]

I am trying to debug a crash in our Fortran / C++ application.

Fortran compiler is
12:17 $ /opt/intel/bin/ifort --version
ifort (IFORT) 15.0.6 20151119
Copyright (C) 1985-2015 Intel Corporation.  All rights reserved.

C++ compiler is:
12:17 $ gcc --version
gcc (Ubuntu 4.9.3-8ubuntu2~14.04) 4.9.3

When I try to run my application under the debugger (either Ubuntu native gdb or gdb-ia), I get the following error:
12:18 $ /opt/intel/bin/gdb-ia ./my-app
No symbol table is loaded.  Use the "file" command.
GNU gdb (GDB) 7.8-8.0.137 *
[...]
Reading symbols from ./grasp-analysis...Segmentation fault (core dumped)

To analyze the problem I ran the gdb-ia debugger under the debugger..., ie:
$ /usr/bin/gdb --args gdb-ia my-application
[...]
Reading symbols from /home/jl/work/eclAM_Grasp/grasp_obj_debug/grasp-analysis...
Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106    ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) bt
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x0000000000687cab in add_partial_symbol ()
#2  0x0000000000687eaf in add_partial_module ()
#3  0x000000000068736a in scan_partial_symbols ()
#4  0x00000000006859e2 in process_psymtab_comp_unit_reader ()
#5  0x0000000000684d1a in init_cutu_and_read_dies ()
#6  0x0000000000685e82 in process_psymtab_comp_unit ()
#7  0x0000000000686b0b in dwarf2_build_psymtabs_hard ()
#8  0x00000000006818ce in dwarf2_build_psymtabs ()
#9  0x0000000000555ce5 in read_psyms ()
#10 0x00000000005e46c5 in require_partial_symbols ()
#11 0x00000000005ea4a9 in read_symbols ()
#12 0x00000000005ea9eb in syms_from_objfile_1 ()
#13 0x00000000005eaa2e in syms_from_objfile ()
#14 0x00000000005eac2c in symbol_file_add_with_addrs ()
#15 0x00000000005eae21 in symbol_file_add_from_bfd ()
#16 0x00000000005eae7c in symbol_file_add ()
#17 0x00000000005eaf09 in symbol_file_add_main_1 ()
#18 0x00000000005eaeb7 in symbol_file_add_main ()
#19 0x000000000061fc4b in catch_command_errors_const ()
#20 0x0000000000620cf2 in captured_main ()
#21 0x000000000061bea2 in catch_errors ()
#22 0x0000000000621075 in gdb_main ()
#23 0x000000000047190f in main ()

Are there any known bug fixes for this error?
Thanks in advance,
Jacob.

[Georg_Z_Intel]

Hello Jacob,

thank you very much for providing the callstack. This helped our engineers to circle the location likely causing the fault. They'll take a closer look.

The root cause is not known yet. It can be the compiler emitting incorrect symbol/debug information or the debugger failing at parsing those.

Hence I'd like to ask you to try:

- The latest 16.0.3 compiler with the gdb-ia from 15.0.6
- Using the latest gdb-ia from 16.0.3 compiler but actually compiling with 15.0.6

The reason for that: We've recently improved debug information emitting & handling in both the latest compiler and debugger versions. I'd like to avoid analyzing an already fixed issue.

Maybe you would be able to provide a small binary showing the problem to aid in finding the root cause and validate the fix on our end. Please share, if you can.

Thank you & best regards,

Georg Zitzlsberger

[Georg_Z_Intel]

Hello Jacob,

One additional thing:

If you just observe this fault with a Fortran application, can you check whether you used BLOCK DATA, e.g.:

      BLOCK DATA

! ABOVE DECLARED WITHOUT A NAME.

!      INTEGER A,B,C

!      REAL    I,J,K,L

!      COMMON  /AREA1/ A,B,C

!      COMMON  /AREA2/ I,J,K,L

!      DATA    A,B,C,I,J,K,L/0,1,2,10.0,-20.0,30.0,-40.0/

! HERE GOES THE DATA DECLARATION.

     END

This is known to cause problems with the current GDB versions, such as gdb-ia. A fix for that will be made available with the gdb-ia 7.10 version released with the 2017 compilers later this year. The current BETA versions don't contain the fix yet.

Best regards,

Georg Zitzlsberger

[Jacob_L_]

Hi

1) Following some links on stackoverflow I also tried a number of different versions of stock GDB. Gdb 7.5.1 seems to accept the file without crashing. This way at least I could debug my problem, although I got some other odd crashes during my debug session so it's no real work-around.

2) Grep'ing for "BLOCK DATA" reveals one file:

      BLOCK DATA
      INTEGER :: IPRINT, LP, MAXFUN, MODE, NFUN
      COMMON /VA13BD/ IPRINT,LP,MAXFUN,MODE,NFUN
      DATA IPRINT/1/,LP/1/,MAXFUN/0/,MODE/1/
      END

I'll try out the latest gdb from 16.0.3 when I get to work tomorrow morning,

/Jacob.

[Jacob_L_]

Tried the gdb from latest version of Intel's compiler. No luck:

12:46 $ /opt/intel-16/bin/gdb-ia my-application
No symbol table is loaded.  Use the "file" command.
GNU gdb (GDB) 7.8-16.0.768
Copyright (C) 2014 Free Software Foundation, Inc; (C) 2013-2016 Intel Corp.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For information about how to find Technical Support, Product Updates,              
User Forums, FAQs, tips and tricks, and other support information, please visit:
<http://www.intel.com/software/products/support/>.For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from my-application...Segmentation fault (core dumped)

Trying the stack-trace trick again reveals:

12:52 $ /opt/intel-16/bin/gdb-ia --args /opt/intel-16/bin/gdb-ia my-application
No symbol table is loaded.  Use the "file" command.
GNU gdb (GDB) 7.8-16.0.768
[...]
Reading symbols from /opt/intel-16/bin/gdb-ia...done.
(gdb) run
Starting program: ....
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
No symbol table is loaded.  Use the "file" command.
GNU gdb (GDB) 7.8-16.0.768
Copyright (C) 2014 Free Software Foundation, Inc; (C) 2013-2016 Intel Corp.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For information about how to find Technical Support, Product Updates,              
User Forums, FAQs, tips and tricks, and other support information, please visit:
<http://www.intel.com/software/products/support/>.For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ....
Catchpoint -2 (signal SIGSEGV), 0x00007ffff6d3e9da in strlen () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0  0x00007ffff6d3e9da in strlen () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x000055555582a008 in add_partial_symbol ()
#2  0x000055555582a47b in scan_partial_symbols ()
#3  0x000055555582ad10 in process_psymtab_comp_unit_reader ()
#4  0x0000555555823291 in init_cutu_and_read_dies ()
#5  0x0000555555825f72 in process_psymtab_comp_unit ()
#6  0x0000555555835ea9 in dwarf2_build_psymtabs ()
#7  0x000055555579a94e in require_partial_symbols ()
#8  0x000055555579f8af in symbol_file_add_with_addrs ()
#9  0x000055555579fdf4 in symbol_file_add ()
#10 0x000055555579fe58 in symbol_file_add_main_1 ()
#11 0x00005555557ca4a2 in catch_command_errors_const.constprop ()
#12 0x00005555557cb0c5 in captured_main ()
#13 0x00005555557c75fa in catch_errors ()
#14 0x00005555557cb75f in gdb_main ()
#15 0x000055555566cda8 in main ()

I don't really know if / how I can build a smaller program example that exhibits the problem, not really being a Fortran programmer myself (I'm on the C++ part of the application)

/Jacob.

[Georg_Z_Intel]

Hello Jacob,

Thank you for testing. This is what I expected. 7.10 shipped with 2017 compilers will have this fixed - engineering just today worked on this.

They also gave me a workaround:
Please try to add a name to the BLOCK DATA in your code. If it's not anonymous it should not trigger the bug in GDB. That would prove talking about the same problem, too. There's still a risk that your problem is different to our assumption.

Best regards,

Georg Zitzlsberger

[Jacob_L_]

Confirmed. I added a name to the single anonumous BLOCK DATA i could find in the source. Recompile, debug now does not crash when reading symbols; this applies both to GNU gdb (GDB) 7.8-8.0.137 and GNU gdb (GDB) 7.8-16.0.768

I'll have to talk to a colleague about the implications of this change, though; thanks for the work-around.

/Jacob.

[Georg_Z_Intel]

Thank you Jacob for quickly validating the workaround! I'll tell the engineers that this was the right guess.

Best regards,

Georg Zitzlsberger