segfault with reshaping and transposing big arrays

[wim_van_hoydonck]

Hi,

For big arrays ( > 1048 by 3000), the following line gives a segfault (complete test program at the end):

b = reshape( transpose(a) , shape(b) )

whereas splitting this line in two, does not result in a crash:

at = transpose(a)
b = reshape( at , shape(b) )

This happens with:
ifort (IFORT) 10.1 20080112
and
ifort (IFORT) 10.1 20070913

with or without optimization.

Compiling with gfortran or g95 doesn't result in a segfault.

For integer(1) arrays, the segfault occurs when the array is approximately 4 times as big. Memory problem?

valgrind reports this (amongst other things):
==5654== Invalid write of size 1
==5654== at 0x402FA0: MAIN__ (test.f90:15)
==5654== Address 0x7FE48E380 is on thread 1's stack
==5654== Can't extend stack to 0x7FE48DA90 during signal delivery for thread 1:
==5654== no stack segment
==5654==
==5654== Process terminating with default action of signal 11 (SIGSEGV)
==5654== Access not within mapped region at address 0x7FE48DA90
==5654== at 0x402FA0: MAIN__ (test.f90:15)
==5654==
==5654== Invalid write of size 8
==5654== at 0x4802338: _vgnU_freeres (vg_preloaded.c:56)
==5654== Address 0x7FE48E2F8 is on thread 1's stack
==5654==
==5654== Process terminating with default action of signal 11 (SIGSEGV)
==5654== Access not within mapped region at address 0x7FE48E2F8
==5654== at 0x4802338: _vgnU_freeres (vg_preloaded.c:56)



Greetings,

Wim


!*test.f90************************************
program test

integer , parameter :: nx = 1049 , ny = 3000
integer , allocatable :: a(:,:) , at(:,:) , b(:)
integer :: status

allocate ( a(nx,ny) , b(nx*ny) , at(ny,nx) , stat=status )
print *, status

forall(i=1:nx,j=1:ny) a(i,j) = i*j

at = transpose(a)
b = reshape( at , shape(b) )

! ifort segfaults here
b = reshape( transpose(a) , shape(b) )

deallocate( a , b , at )

end program test
!********************************************************

[oh_moose]

Looks like a shameless stack abuse case to me.
Try to compile your software with the qualifier -heap-arrays 0

[wim_van_hoydonck]

Thanks,

That works :)

Now I have to figure out why...

[oh_moose]

On Linux/x86 the stack size is limited to 8 MiB. That is a pretty large stack for calling routines. Unfortunately some languages put the actual values of function arguments onto the stack for a function call. Fortran puts references (pointers) to the arguments onto the stack. And to make matters worse, some compilers put all local variables of functions onto the stack, too. At least the Intel Fortran compiler for Linux allows you to set the maximum size of arrays on the stack. Any array above the specified size ends up on the heap (see woman ifort). With -heap-arrays 0 you tell the compiler to put all arrays (including temporary copies and intermediate results) onto the heap. I am sure there is a performance penalty for doing the right thing. You could tune this option, but with large arrays on the stack you may just postpone the crash.

[Steven_L_Intel1]

My understanding is that on Linux the stacksize is initially set by a kernel parameter. You can raise the stacksize limit with commands such as:

ulimit -s
or
limit stacksize unlimited

where "unlimited" really means "maximum as defined when the kernel was built". The stack can't exceed 1GB on 32-bit, I believe.

Regarding the -heap-arrays switch - you can leave off the 0 as that is the default. There is a small performance penalty due to the allocate and deallocate calls but my feeling is that this is typically swamped by the routine execution time. You can specify a non-zero parameter, so that allocations under the specified size are still done on the stack, but this affects only those cases where the size is known at compile-time.

If using -heap-arrays means that the program runs whereas without it it doesn't, the performance difference is not relevant.

[wim_van_hoydonck]

Thanks for the answers.

Stack size was 10240 kbytes (Fedora 8 x86_64). Setting it to unlimited (ulimit -s unlimited) solved the problem, as does the switch -heap-arrays.

Greetings,

Wim

[kristipati]

Greetings,

i have a similar problem. I am getting the following error when I compile my code with ifort with optimization -03. It is a fortran code.

I re-ran with unlimited stack size but, this time, the problem fails for larger arrays (larger grids).

Can't extend stack to 0x7FE7BE2B0 during signal delivery for thread 1:
==2255== no stack segment
==2255==
==2255== Process terminating with default action of signal 11 (SIGSEGV)
==2255== Access not within mapped region at address 0x7FE7BE2B0
==2255== at 0x42E7C3: global_mp_read_data_

When I ran with using -heap-arrays 0, it does not compile and throws an error. What is the correct way of using this option -heap-arrays 0 ?

$ mpif77 -O3 -heap-arrays 0 -o 500iterV0.exe V0.f90
ld: 1: No such file: No such file or directory
mpif77: No such file or directory

$ mpif77 -heap-arrays 0 -O3 -o 500iterV0.exe V0.f90
ld: 0: No such file: No such file or directory
mpif77: No such file or directory

I know fortran compiler exists as I am able to compile the code without using -heap-arrays 0 as shown below:

$ mpif77 -O3 -o 500iterV0.exe V0.f90

Pl advise. Thank you.

Quoting - wim.van.hoydonck@gmail.com

Thanks for the answers.

Stack size was 10240 kbytes (Fedora 8 x86_64). Setting it to unlimited (ulimit -s unlimited) solved the problem, as does the switch -heap-arrays.

Greetings,

Wim

[kristipati]

OS Info:

Linux 2.6.12.6 #1 Thu Dec 22 19:30:31 EST 2005 x86_64 x86_64 x86_64 GNU/Linux

[Steven_L_Intel1]

Just leave off the 0.

[kristipati]

I did that and I was able to compile the code.

But, when I run the valgrind, I still get the stack error.

$ mpif77 -heap-arrays -O3 -o 500iterV0.exe V0.f90

$ mpirun -np 1 valgrind --tool=cachegrind ./500iterV0.exe
==6685== Cachegrind, an I1/D1/L2 cache profiler.
==6685== Copyright (C) 2002-2005, and GNU GPL'd, by Nicholas Nethercote et al.
==6685== Using LibVEX rev 1471, a library for dynamic binary translation.
==6685== Copyright (C) 2004-2005, and GNU GPL'd, by OpenWorks LLP.
==6685== Using valgrind-3.1.0, a dynamic binary instrumentation framework.
==6685== Copyright (C) 2000-2005, and GNU GPL'd, by Julian Seward et al.
==6685== For more details, rerun with: -v
==6685==
g0_1.dat
==6685== Can't extend stack to 0x7FE7BE000 during signal delivery for thread 1:
==6685== no stack segment
==6685==
==6685== Process terminating with default action of signal 11 (SIGSEGV)
==6685== Access not within mapped region at address 0x7FE7BE000
==6685== at 0x42E7C3: global_mp_read_data_ (in /scratch/pavan/thesis/600/500iterV0.exe)
==6685==
==6685== I refs: 958,292
==6685== I1 misses: 3,741
==6685== L2i misses: 3,092
==6685== I1 miss rate: 0.39%
==6685== L2i miss rate: 0.32%
==6685==
==6685== D refs: 368,659 (248,427 rd + 120,232 wr)
==6685== D1 misses: 17,562 ( 10,986 rd + 6,576 wr)
==6685== L2d misses: 11,220 ( 5,126 rd + 6,094 wr)
==6685== D1 miss rate: 4.7% ( 4.4% + 5.4% )
==6685== L2d miss rate: 3.0% ( 2.0% + 5.0% )
==6685==
==6685== L2 refs: 21,303 ( 14,727 rd + 6,576 wr)
==6685== L2 misses: 14,312 ( 8,218 rd + 6,094 wr)
==6685== L2 miss rate: 1.0% ( 0.6% + 5.0% )
-----------------------------------------------------------------------------
One of the processes started by mpirun has exited with a nonzero exit
code. This typically indicates that the process finished in error.
If your process did not finish in error, be sure to include a "return
0" or "exit(0)" in your C code before exiting the application.

PID 6685 failed on node n0 (127.0.0.1) due to signal 11.
-----------------------------------------------------------------------------

Quoting - Steve Lionel (Intel)

Just leave off the 0.