Runtime Difference in allocation Codes

[bluequartz]

I was recently debugging some codes and I ran into an issue where compiled code works as intended on Windows with v19u5 but crashes on macOS v19u5. I was looking for some feedback regarding our specific use of the Fortran code in question and why the compiled code would work on one platform but not another. Here is the first bit of offending code: recursive function HDF_writeDatasetIntegerArray2D(dataname, intarr, dim0, dim1, HDF_head, overwrite) result(success) !DEC$ ATTRIBUTES DLLEXPORT :: HDF_writeDatasetIntegerArray2D use ISO_C_BINDING IMPLICIT NONE character(fnlen),INTENT(IN) :: dataname integer(kind=irg),INTENT(IN) :: dim0 integer(kind=irg),INTENT(IN) :: dim1 integer(kind=irg),INTENT(IN) :: intarr(dim0, dim1) type(HDFobjectStackType),INTENT(INOUT) :: HDF_head !f2py intent(in,out) :: HDF_head logical,INTENT(IN),OPTIONAL :: overwrite integer(kind=irg) :: success integer(HID_T) :: space, dset ! Handles integer :: hdferr, rnk integer(HSIZE_T), DIMENSION(1:2) :: dims integer, dimension(1:dim0,1:dim1), TARGET :: wdata TYPE(C_PTR) :: f_ptr success = 0 dims(1:2) = (/ dim0, dim1 /) wdata = intarr <====== DIES HERE And here is the corrected code: recursive function HDF_writeDatasetIntegerArray2D(dataname, intarr, dim0, dim1, HDF_head, overwrite) result(success) !DEC$ ATTRIBUTES DLLEXPORT :: HDF_writeDatasetIntegerArray2D use ISO_C_BINDING IMPLICIT NONE character(fnlen),INTENT(IN) :: dataname integer(kind=irg),INTENT(IN) :: dim0 integer(kind=irg),INTENT(IN) :: dim1 integer(kind=irg),INTENT(IN) :: intarr(dim0, dim1) type(HDFobjectStackType),INTENT(INOUT) :: HDF_head !f2py intent(in,out) :: HDF_head logical,INTENT(IN),OPTIONAL :: overwrite integer(kind=irg) :: success, istat integer(HID_T) :: space, dset ! Handles integer :: hdferr, rnk integer(HSIZE_T), DIMENSION(1:2) :: dims integer(kind=4),allocatable,TARGET :: wdata(:,:) TYPE(C_PTR) :: f_ptr success = 0 allocate(wdata(dim0, dim1), stat=istat) wdata = intarr dims(1:2) = (/ dim0, dim1 /) Note that in the "working" version we manually do the allocation for the wdata (later we deallocate). This working code now runs correctly. It is my understanding that either _should_ have worked. And in fact when we try to reduce it to a simple test cast the original code does work correctly which is why we couldn't submit anything as a bug. So there is clearly something else in our environment that might be causing the issue. Third party libraries? Link arguments, something else? I would be appreciative of any ideas of why our code needed the change. Many Thanks Mike Jackson

[FortranFan]

@Mike Jackson,

What are the values of dim0, dim1?  Are they big relative to the stack size and other OS and hardware-specific limits with your macOS v19u5?

In your first case, you've an automatic array in wdata and program may be placing it on the stack and running into size limitations resulting in the program crash.

Take a look at this: https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference-heap-arrays and see if that helps.

[bluequartz]

Dim0 = 501 and Dim2 = 501 which for a 4 byte integer is about 1MB of data. That should be doable on macOS. I know I do that sort of allocation in C/C++ all the time without out issues (but that is a difference language).

I'll take a look at the link. Thanks for sending that along.

On Windows since it by default has a very small stack size we manually increase it to 100MB which maybe why we don't see the issue over on that platform. I have never really run into stack size issues on macOS or Linux so I never really think about it.

--

Mike Jackson

[jimdempseyatthecove]

In your failing code, insert as your first statement

print *,dim0,dim1

IOW do not rely on the caller's values for Dim0 and Dim1

If you error out before the printout, then suspect an argument passing error.

You can also insert the print into the working copy. Larger array allocation may succeed and may be usable by remainder of code.

Jim Dempsey

[bluequartz]

Dear Jim,

  In the failing code we couldn't get a printout to work (write(*,*)'Dim0=',dim0) as it would crash on that line also. I was suspecting stack corruption for a while. I just came across the -heap-arrays compiler argument which is probably what we needed all along. I found it odd that the code would work on one platform and not the other (macOS vs Windows) with seemingly the same compiler arguments.

Mike Jackson

[jimdempseyatthecove]

>> I found it odd that the code would work on one platform and not the other (macOS vs Windows) with seemingly the same compiler arguments.

The stack organization is different and if macOS is similar to Linux, Linux generates PIC code format (Position Independent). IOW the memory layout is different. I.E. different layout of code.

In the failing code, if you swap the declarations of dims and wdata, does the code then fail on dims(1:2) = (/ dim0, dim1 /)?
IOW place the memory location of dims after (below on stack) wdata.

If so, then this would be indicative of stack size issue (assuming compiler kept stack order in declaration order). If you can conclusively determine this is a stack size issue then you have something in your control to fix. Perhaps your assumption on how to specify stack size for macOS is not correct.

Jim Dempsey