Segfault on load of

Joseph_Thames · ‎09-24-2010

Hi

We just intalled the ifort 11.1-072 compiler and were testing a program we had running with gfortran. This program failed to load, receiving a segfault. So we couldn't load it with the Intel debugger. When we used gdb and valgrind to attempt to locate the place where the segfault occurred, they both identied address 0x2:

Here is the gdb output

[tosva@centosva32 woodzh]$ gdb /usr/local/MC7Cbin/v4.00/fctran

GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5_5.2)

License GPLv3+: GNU GPL version 3 or later <>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law. Type "show copying"

and "show warranty" for details.

This GDB was configured as "i386-redhat-linux-gnu".

For bug reporting instructions, please see:

<>...

Reading symbols from /usr/local/MC7Cbin/v4.00/fctran...(no debugging symbols found)...done.

(gdb) run woodzh.fc

Starting program: /usr/local/MC7Cbin/v4.00/fctran woodzh.fc

Program received signal SIGSEGV, Segmentation fault.

0x00000002 in ?? ()

(gdb) bt

#0 0x00000002 in ?? ()

Here is the Valgrind output:

[tosva@centosva32 woodzh]$ valgrind -v /usr/local/MC7Cbin/v4.00/fctran woodzh.fc

==12778== Memcheck, a memory error detector

==12778== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info

==12778== Command: /usr/local/MC7Cbin/v4.00/fctran woodzh.fc

==12778==

--12778-- Valgrind options:

--12778-- -v

--12778-- Contents of /proc/version:

--12778-- Linux version 2.6.18-194.el5 (mockbuild@builder16.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Apr 2 14:58:35 EDT 2010

--12778-- Arch and hwcaps: X86, x86-sse1-sse2

--12778-- Page sizes: currently 4096, max supported 4096

--12778-- Valgrind library directory: /usr/lib/valgrind

--12778-- Reading syms from /usr/local/MC7Cbin/v4.00/fctran (0x108000)

--12778-- Reading syms from /usr/lib/valgrind/memcheck-x86-linux (0x38000000)

--12778-- object doesn't have a dynamic symbol table

--12778-- Reading suppressions file: /usr/lib/valgrind/default.supp

==12778== Jump to the invalid address stated on the next line

==12778== at 0x2: ???

==12778== Address 0x2 is not stack'd, malloc'd or (recently) free'd

==12778==

==12778== Process terminating with default action of signal 11 (SIGSEGV)

==12778== Bad permissions for mapped region at address 0x2

==12778== at 0x2: ???

==12778==

==12778== HEAP SUMMARY:

==12778== in use at exit: 0 bytes in 0 blocks

==12778== total heap usage: 0 allocs, 0 frees, 0 bytes allocated

==12778==

==12778== All heap blocks were freed -- no leaks are possible

==12778==

==12778== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

==12778==

==12778== 1 errors in context 1 of 1:

==12778== Jump to the invalid address stated on the next line

==12778== at 0x2: ???

==12778== Address 0x2 is not stack'd, malloc'd or (recently) free'd

==12778==

==12778== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Segmentation fault

Since the 0x2 address seemed to suggest that some initialization code was not being loaded, we decided to compare the gfortran loadmap with the ifort loadmap using Meld, to see what was different. Here is the result:

Meld image of loadmaps comparison

Note the missing load of crt1.o, which apparently is supposed to invoke a call to main(). Is this the source of the segfault?

How do I get around this problem?

Thanks,

Joseph Thames

beartham@gmail.com

mecej4 · ‎09-24-2010

Before trying to analyze the dump traces from a large program, with a new installation of IFort it would be worthwhile to compile and run some small programs. For example:

program tst
integer :: i,j
do i=1,10
j=i*i
write(*,*)i,j
end do
end program tst

Joseph_Thames · ‎09-25-2010

Ok, I ran your nice little program, getting the results I expected:

[tosva@centosva32 ~]$ ./a.out
1 1
2 4
3 9
4 16
5 25
6 36
7 49
8 64
9 81
10 100

THIS IS NOT A BUG IN MY PROGRAM!!!!! It has been running for many years using various fortran compilers, including g77 and gfortran.

This program (fctran) is a translator for a higher-level modeling language which generates a shared-object executable that interfaces with a large shared-object library of optimization tools. I have recently run into this same bug in attempting to load generated test programs when a static library of these tools was used.

Static library versions of this software were fully functional on Suse 9.0 in 2005 with the g77 compiler and on Fedora 8 in 2007 with the gfortran compiler.

But on Unbuntu 8.04, 9.04, and 9.10, CentOS 5.5 and Scientific Linux 5.5, I was never able to get the static library version of any of the fctran-generated programs to successfully load. In each case the result was a segfault. But fctran itself (a standalone program) never had a problem (until now).

I was finally able to get several generated test programs (shared objects) to load and run successfully when I converted the library to a shared-object library. But then I would sometimes get segfaults, well into their execution, at wierd places like END statements and CLOSE statements in the generated fortran. All of these segfaults were in glibc routines, usually associated with malloc.

Now, since my installed Intel fortran also uses glibc, I strongly suspect this is also the source of this problem. In fact, I suspect that it is in the routine that prints the GNU banner at the very beginning of the execution, before any fortran executable statement is executed.

My take on this issue is that the GNU loader (ld) and c libraries have become downgraded in reliability since 2007, and I would like to see Intel set up some rigorous regression tests to find and fix these bugs.

The specific question raised in my original post is why is the Intel version not loading the crt1.o routine, which is what calls main()?

Thanks,

Joseph Thames

PS regarding the original post: to link to the Meld image, you need to remove the Intel prefix to the url to get www.metacalculus.com/misc/.

TimP · ‎09-25-2010

As you're raising a disjointed group of issues, I'll continue in the same vein.
ifort 11.1.072 was replaced fairly quickly by 11.1.073, I think partly to adjust support for SuSE and Ubuntu.
glibc itself has been moving away from support of static libraries for years; I doubt Intel has much influence in that, but I don't know that static glibc could get much testing along with ifort. Static libraries aren't even available on current Red Hat install DVD.
glibc also has been moving away from supporting backward compatibility as far back as Intel compilers have been doing. I think the focus has to shift to Ubuntu 10.04, Red Hat 5.5 and 6.0, SLES 11, ..., in versions under test. I don't think there's much testing of near-clones of Red Hat, if you don't see them called out specifically in the compatibility list. Some Intel software products are supporting primarily x86_64, since few 32-bit-only CPUs remain in use, and 64-bit OS installations outnumber 32-bit even on the web sales sites.

Lorri_M_Intel · ‎09-27-2010

Would you mind posting the command line you used to build your application please?

There are a small number of *very unusual* circumstances where we wouldn't list crt1.o but generally speaking, yes, it is included in the resulting executable.

thank you --

-- Lorri

Joseph_Thames · ‎09-27-2010

Hi tim18,

I understand about the static libraries. We have moved away from them as well. But the fctran program is an executable module totally compiled from source. Thus It only accesses standard libraries. Yet it won't load when compiled with ifort, even though it runs with no errors when compiled with g77 or gfortran.

We have converted all our static libraries to shared object libraries (with g77, gfortran, and ifort). We have tested these libraries with g77 and gfortran with our regression test suite, but many tests encounter segfaults in glibc routines. We recently moved to ifort, in the hope of better reliability, but since it also depends on glibc, and our first test (of fctran) failed to load, it seems we are back in the same swamp we have been in for months, even with Intel products.

We are using CentOS 5.5 beacuse we are converting to client-server mode, and that is what our VPS vendor supports. We moved away from our Ubuntu 8.04 VPS because of the reliability issue, hoping to get better reliabillity with a near RHEL clone.

We have a x86_64 CentOS 5.5 VPS as well, and our intent is to move to 64-bit architecture soon, but after we have all our regression tests working. Although this SW once ran on a Cray YMP, it is not trivial to convert to 64-bit. We want to do it coming from a stable 32-bit version.

We have the Intel C/C++ compiiler, though we haven't tested it yet. Is there a way to install a verstion of ifort that does not depend on glibc?

Thanks,

Joseph Thames

jimdempseyatthecove · ‎09-27-2010

Have you tried explicityly linking with crt1.o?

Is this a mixed language app?

If so, are you using a C/C++ main or Fortran PROGRAM as entry?

If Fortran PROGRAM, do you have static C++ ctors?

When writting a dummy test program, try to make it configured as the failing program (same entry, same libraries, etc...). What you are looking for is an easy reproducer. This is most likely a case of file ommission.

Jim Dempsey

TimP · ‎09-27-2010

I agree that CentOS is an important platform for customers using ifort; I only meant to point out that it's not listed as a tested OS for ifort. If you find a specific reproducible issue with ifort and glibc in CentOS, you should still submit it on your premier.intel.com account, as otherwise everyone is simply relying on the assertion of CentOS compatibility with Red Hat and hoping that your issues are covered in routine testing with Red Hat.
ifort for linux is tested only on glibc platforms, as far as I know. If you had a g++ installation based on another library, it would use that library. Unfortunately, it's not likely that another library is tested as well as glibc.
Portable Fortran should not be difficult to convert to x86_64. 64-bit integers are desirable in some contexts, but not mandatory.