code producing segmentation fault on offload with -openmp option

aketh_t_ · ‎10-26-2015

Hi all

My code has a module state_test which makes a call to state.

On offloading the call to state I get a segfault.

Here is the call to state

!dir$ offload begin target(mic:0)in(TRCR)out(RHOK1,RHOK2,RHOK3,RHOK4)
call state(k,kk,TRCR(:,:,:,1), TRCR(:,:,:,2), this_block,RHOK1,RHOK2,RHOK3,RHOK4)
!dir$ end offload

However when offload is performed without the -openmp option it works fine.

Here is the compile line that segfaults.

ifort state_test.F90 state_mod.F90 -openmp

[Offload] [HOST]          [State]           Initialize logical card 0 = physical card 0
[Offload] [HOST]          [State]           Initialize logical card 1 = physical card 1
[Offload] [MIC 0] [File]                    state_test.F90
[Offload] [MIC 0] [Line]                    22
[Offload] [MIC 0] [Tag]                     Tag 0
[Offload] [HOST] [Tag 0] [State]           Start target
[Offload] [HOST] [Tag 0] [State]           Setup target entry: __offload_entry_state_test_F90_22MAIN__ifort0101298585939NpI8VK
[Offload] [HOST] [Tag 0] [State]           Host->target pointer data 0
[Offload] [HOST] [Tag 0] [Signal]          signal : none
[Offload] [HOST] [Tag 0] [Signal]          waits : none
[Offload] [HOST] [Tag 0] [State]           Host->target pointer data 23708160
[Offload] [HOST] [Tag 0] [State]           Host->target copyin data 52
[Offload] [HOST] [Tag 0] [State]           Execute task on target
[Offload] [HOST] [Tag 0] [State]           Target->host pointer data 31610880
[Offload] [MIC 0] [Tag 0] [State]           Start target entry: __offload_entry_state_test_F90_22MAIN__ifort0101298585939NpI8VK
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr_MAIN__.30 NOCOPY
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr_state_dummy_$TRCR_V$1e.0 IN
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr_MAIN__.30 OUT
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr_MAIN__.30 OUT
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr_MAIN__.30 OUT
[Offload] [MIC 0] [Tag 0] [Var]             __offload_stack_ptr_MAIN__.30 OUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$K_V$22 INOUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$KK_V$20 INOUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$THIS_BLOCK_V$21 INOUT
here
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
a.out              0000000000475591 Unknown               Unknown Unknown
a.out              0000000000473CE7 Unknown               Unknown Unknown
a.out              00000000004441A4 Unknown               Unknown Unknown
a.out              0000000000443FB6 Unknown               Unknown Unknown
a.out              0000000000426F34 Unknown               Unknown Unknown
a.out              0000000000405A4D Unknown               Unknown Unknown
libpthread.so.0    000000328AC0F500 Unknown               Unknown Unknown
libcoi_host.so.0   0000003AB0436F97 Unknown               Unknown Unknown
libcoi_host.so.0   0000003AB0438114 Unknown               Unknown Unknown
libcoi_host.so.0   0000003AB043821A Unknown               Unknown Unknown
libcoi_host.so.0   0000003AB0439149 Unknown               Unknown Unknown
libpthread.so.0    000000328AC07851 Unknown               Unknown Unknown
libc.so.6          000000328A4E890D Unknown               Unknown Unknown

Here is the compile line that works perfectly

ifort state_test.F90 state_mod.F90

[Offload] [HOST]          [State]           Initialize logical card 0 = physical card 0
[Offload] [HOST]          [State]           Initialize logical card 1 = physical card 1
[Offload] [MIC 0] [File]                    state_test.F90
[Offload] [MIC 0] [Line]                    22
[Offload] [MIC 0] [Tag]                     Tag 0
[Offload] [HOST] [Tag 0] [State]           Start target
[Offload] [HOST] [Tag 0] [State]           Setup target entry: __offload_entry_state_test_F90_22MAIN__ifort0102086235354lztzYz
[Offload] [HOST] [Tag 0] [State]           Host->target pointer data 0
[Offload] [HOST] [Tag 0] [Signal]          signal : none
[Offload] [HOST] [Tag 0] [Signal]          waits : none
[Offload] [HOST] [Tag 0] [State]           Host->target pointer data 0
[Offload] [HOST] [Tag 0] [State]           Host->target copyin data 23708172
[Offload] [HOST] [Tag 0] [State]           Execute task on target
[Offload] [HOST] [Tag 0] [State]           Target->host pointer data 0
[Offload] [MIC 0] [Tag 0] [State]           Start target entry: __offload_entry_state_test_F90_22MAIN__ifort0102086235354lztzYz
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$TRCR_V$1e IN
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$RHOK4_V$1a OUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$RHOK3_V$1b OUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$RHOK2_V$1c OUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$RHOK1_V$1d OUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$K_V$22 INOUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$KK_V$20 INOUT
[Offload] [MIC 0] [Tag 0] [Var]             state_dummy_$THIS_BLOCK_V$21 INOUT
here
in state
0.000000000000000E+000
   52422.4740678280
   35057.7950417253
   83.6184972370540
   37.7420169087246
   2.67158392565487
   13963.8106092726
   23339.2149038234
   23339.2149038234
ended state
[Offload] [MIC 0] [Tag 0] [State]           Target->host copyout data   31610892
[Offload] [HOST] [Tag 0] [CPU Time]        0.373570(seconds)
[Offload] [MIC 0] [Tag 0] [CPU->MIC Data]   23708172 (bytes)
[Offload] [MIC 0] [Tag 0] [MIC Time]        0.232588(seconds)
[Offload] [MIC 0] [Tag 0] [MIC->CPU Data]   31610892 (bytes)

Why does this happen?

I tried with various values of PHI_OMP_STACKSIZE and OMP_STACKSIZE from 1M,2M,4M,64M,128M to 50G.

It still fails why is that so?

Attached also is the code.

jimdempseyatthecove · ‎10-26-2015

Is your offload region inside an OpenMP parallel region?

Jim Dempsey

aketh_t_ · ‎10-26-2015

No i am not spawning any threads right now atleast, but I plan to.

There are absolutely no omp sections right now.

I do wish to create on later.

aketh_t_ · ‎10-26-2015

is the problem with ulimit -s.

i.e stacksize on MIC.

I am unable to find a way to change the stacksize on MIC with ulimit permenentaly.

I guess that may the problem.

TimP · ‎10-26-2015

Offload has a separate environment variable MIC_STACKSIZE (not controlled by MIC_ENV_PREFIX). It looks like you didn't alter the stack limit you intended.

A long time ago, it was said there would be consideration of a permanent way to change MIC default ulimit, but I haven't heard of any outcome. This would not necessarily change the default for offload. Due to the lack of a permanent way to set it, there are several situations in which it may be best to embed your application in a script which can include the appropriate settings.

In general, under OpenMP, you must give attention both to ulimit -s and OMP_STACKSIZE, increases in the latter likely requiring adjustment of the former. This doesn't necessarily apply to offload mode.