Stack overflow

[Horsewood__Jerry]

While this forum has numerous entries on the topic of stack overflow, none seem to match the situation I am encountering. I am developing a multi-language Windows application (32-bit) consisting of a C# GUI and several dll's of Fortran procedures that provide a variety of features under the umbrella of the GUI. The development environment consists of Visual Studio 2015 and the IVF 18.0 compiler. The operating system is Windows 10 on a 64-bit computer.

One of the features initiates an iterative process that takes an unknown amount of time to complete. The feature is initiated with a call from the GUI to a subroutine in one of the dll's. Because of the time involved, it is run in it's own thread using System.Threading. Once started, the subroutine runs to completion, periodically spawning files of information that is used to update the display of the GUI. The called subroutine was originally the main program of a standalone application with 22 additional modules. The main program was converted to a subroutine contained within its own module for purposes of creating the dll. Upon execution, the thread terminates with a Stack Overflow exception at the call to a subroutine that is embedded within one of the other modules. In the debugger, the exception is raised immediately upon stepping into the subroutine, suggesting that the problem is with the subroutine arguments. The call to the subroutine appears as follows:

call IntervalHams(OptHam, P, .FALSE.)

where P is a double precision scalar and OptHam is an object of a derived type that consists of several double precision scalar variables, followed by several KIND=4 integer and logical scalar variables, and ending with an explicit array of dimension 1860 of another derived type record of scalars totaling 64 bytes. OptHam is of fixed length of 119,264 bytes.

The numerous entries on this forum regarding stack overflow invariably point to instances of automatic or allocated arrays and suggest increasing the stack reserve size or specifying a heap size to force automatic and allocated arrays to be allocated on the heap rather than the stack. I have increased the stack size to ten times the size of the OptHam record, which had no effect. Also, explicitly specifying the heap size has no effect, which was expected since there are no automatic or allocated arrays. Does anyone know of any other possible reason for the exception in this case?

[FortranFan]

Horsewood, Jerry wrote:

With this post, I attempt to summarize what has been learned about the Stack Overflow exception described in previous posts above: ..

4. Collectively, these facts lead me to conclude that there is a serious problem with the IVF compiler. At a minimum, improved diagnostics are needed to identify more specifically what it thinks is wrong and where it initially manifests itself.

@Horsewood, Jerry,

See Steve Lionel's suggestion in Quote #20.  You need to provide appropriate details as suggested by Steve either here or privately to Steve or to Intel Support to resolve your issue.

Consider the following silly code based on your original post and notice the run-time exception involving stack overflow.  This exception does not really indicate "a serious problem with the IVF compiler," rather it can be a situation involving the compiler options.  A different set of options e.g., /heap-arrays0 allows this particular case to run with no exceptions.  Note I'm not suggesting /heap-arrays0 is the magic bullet for you, just that if you can either simplify your Fortran code to a minimal working example as shown here OR share all the relevant details of your code, compiler and linking options, etc. as mentioned above, you have a good chance to solving your problem:

C:\Temp>type main.f90
module t_m
   integer, parameter :: N = 300000
   type :: t
      integer :: x(N)
   end type
end module
subroutine IntervalHams( targ, parg, larg )
   use t_m
   type(t)          :: targ
   double precision :: parg
   logical          :: larg
   if ( .not. larg ) then
      print *, "In OptHam:"
      print *, targ%x(1), "; expected = 42"
   end if
   return
end subroutine
subroutine sub( isiz, iarr )
   use t_m
   integer :: iarr(*)
   integer :: isiz
   type(t) :: OptHam
   double precision :: p
   if (isiz == N ) OptHam%x = iarr(1:siz)
   call IntervalHams(OptHam, P, .false.)
end subroutine
program main
   use t_m, only : N
   integer :: x(2*N)
   x = 42
   call sub( N, x(1:size(x):2) )
   stop
end program

C:\Temp>ifort main.f90
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on IA-32,
Version 18.0.3.210 Build 20180410
Copyright (C) 1985-2018 Intel Corporation.  All rights reserved.

Microsoft (R) Incremental Linker Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:main.exe
-subsystem:console
main.obj

C:\Temp>main
forrtl: severe (170): Program Exception - stack overflow
Image              PC        Routine            Line        Source
main.exe           008B4689  Unknown               Unknown  Unknown
main.exe           008B4443  Unknown               Unknown  Unknown
main.exe           008B4844  Unknown               Unknown  Unknown
kernel32.dll       74C9343D  Unknown               Unknown  Unknown
ntdll.dll          774D9802  Unknown               Unknown  Unknown
ntdll.dll          774D97D5  Unknown               Unknown  Unknown

C:\Temp>

With /heap-arrays0,

C:\Temp>ifort /heap-arrays0 main.f90
Intel(R) Visual Fortran Intel(R) 64 Compiler for applications running on IA-32,
Version 18.0.3.210 Build 20180410
Copyright (C) 1985-2018 Intel Corporation.  All rights reserved.

Microsoft (R) Incremental Linker Version 14.00.24215.1
Copyright (C) Microsoft Corporation.  All rights reserved.

-out:main.exe
-subsystem:console
main.obj

C:\Temp>main
 In OptHam
          42 ; expected = 42

C:\Temp>

[Horsewood__Jerry]

Steve, I am willing to private message you the source code and will do so momentarily. Can you provide information on how to private message you?

Jim, your calculation of the stack size allocated in the 41 loops in chkstk is off by a factor of 10; 41 x 4,096 is 167,936, not 1,679,936. So the amount of stack used in chkstk is not very close to the limit of 2MB. I don't know how to determine what was allocated because the register values do not seem to be visible in the debugger.

FortranFan, the exception raised in your example is caused by the automatic array x, which Jim and Steve have pointed out can cause the exception and can be avoided by allocating on the heap rather than the stack. In my case, the array of OptHam is of fixed dimension (1860). There are no automatic arrays in either the called procedure or the calling procedure, so I don't see that this is likely to be the source of the problem, but who knows. Possibly Steve might be able to find the culprit when he compiles and tests the application.

Please accept my thanks to all who have weighed in on this issue. It is a challenging problem.

[IanH]

If your compile options accomodate recursion (which will be the default one day), all local variables will go on the stack, not just automatics. 

Expression temporaries can also go on the stack.  Expression (and perhaps argument) temporaries are often seen with objects that have the TARGET attribute (which includes anything being referenced by a Fortran pointer).

The default stack allocation on Windows is chronically inadequate. For programs of any reasonable complexity you should expect stack overflow if the size is left at the default.

Some of the content in the opening post perhaps needs clarification.  Objects that are allocated (i.e using an ALLOCATE statement) always go on the heap; specifying a heap size isn't relevant (or useful) - what is typically required is to tell the compiler to put the temporaries being discussed above on the heap.  The compiler option to put temporaries on the heap (/heap-arrays) takes a size argument, but the only practically useful value for this argument is zero. It isn't clear to me that the OP has actually tried /heap-arrays.

Not being able to see register values in the debugger, when the program is at a breakpoint or similar, is rather odd.

Trying to diagnose specifics without seeing code is always going to be challenging. Trying to construct a minimal reproducer often results in discovering the problem and solution along the way.

[Steve_Lionel]

Jerry, I have to initiate the private message conversation, which I have now done. (A perk of being an "Intel Black Belt.") You can respond to me there.

[Horsewood__Jerry]

IanH, thanks for the comments. With regard to compile options, the Fortran/Code Generation property Enable Recursive Routines is set to No. While there probably are expression and argument temporaries created by the code, I don't know what causes them to be generated. I do not use pointers in the code and have not declared any variables with the TARGET attribute.

Your comment about seeing the register values at breakpoints prompted me to look a little closer and found I could see them after declaring them as Watch variables. Referring to the assembly language code listed in Post #17, here is an example of the register values leading up to the failure.

Upon entry of the loop at label CS10 (line 86), the values of the registers are:
ecx = 65,934,248
eax = 66,174,976
esp = 66,175,200
The register eax is always a multiple of 4096. In this case the multiple is 16,156, but I noted it can vary by one or two at different times. The purpose of the loop is to successively reduce eax by 4096 until its value falls below the value of ecx. For this example, 59 passes through the loop would be required. However, in this case the routine works successfully only until the value of eax is 66,002,944. The exception is raised upon executing the 'test' operation at line 99 with this value of eax on the 42nd pass of the 59 expected to reach completion.

As mentioned in previous posts, a call to IntervalHams is executed successfully once in an initialization routine prior to the call that fails. During that call, the values of the registers are similar to above but not exactly the same. There, also, the number of passes to completion is 58 or 59, but no exception is raised. Does this information trigger any new thoughts on a possible solution?

[jimdempseyatthecove]

It would have been best if you obtained eax on entry to _chkstk.

This is the number of bytes to allocate.

This said, at CS10 ecx appears to be the tentative new stack address.

+esp 66175200
-ecx 65934248
=          240952

~241 KB for allocation.

The allocation amount seems reasonable.

Note, if your Linker arguments did not specify sufficient stack size, then the Virtual Memory Page map may mark the page below the requested size as not only not loaded/present, but also as not to be loaded. IOW a guard page between the stack and the program space.

Using 32-bit build, on Windows, "Hello World"

ESP = 0046FDC0 Default stack pointer
ESP = 06E5FC40  Stack reserve 100000000

Upon setting the Stack Reserve to 100 MB, you find that the starting stack pointer has been moved up ~100 MB (111083136).

It still appears that you are not specifying a large enough stack reserve.

Jim Dempsey

[Horsewood__Jerry]

Jim, prior to your last post, I thought the larger stack size you recommended meant a few MB of reserve. I had tried up to 3 MB and the case continued to fail. After seeing your last post, I tried 100 MB and the case ran to completion. Cutting it to 10 MB also works, but 5 MB fails. So I now have a working app and the problem is resolved. Thanks for your help and suggestions. You and Steve are awesome.

[jimdempseyatthecove]

The nomenclature used is a bit confusing, as well as the entry values being in bytes as opposed to KB (not sure if VS accepts suffixes kb, mb).

Reserve means the virtual address space reserved, but not necessarily committed to RAM and/or Page File. Commit occurs during runtime as you touch pages not committed. This is the reason for the 4KB walk in the _chkstk to assure that should there be a failure, that it occurs inside _chkstk as opposed to after return when application touches pages beyond what is permitted for the process.

Commit means that at program load/C Runtime init time that the committed portion of the stack is touched, thus the application will not ab-end or suffer excessive latency during runtime while performing page allocation (and possible wipe) of pages in the page file.

For normal applications you would not use Commit.

Jim Dempsey