0xC0000005 Access Violation Writing Location Error

[emreka82]

Hi,

I am writing a code with large arrays. The code works fine for small matrices but when I go beyond a certain level, the following error pops out:

Unhandled exception at 0x77913560 in 3d.exe: 0xC0000005: Access violation writing location 0x0000000000050e14.

I am dynamically allocating memory with variables, then deallocating them. The code is very long, longer than 20,000 lines. I want to trace the problem. Before deciding to start this topic, I search the forum and other sources. The "Stack Size" problem has seemed reasonable. I changed reserve size to 200,000,000 and commit size 20,000,000. The problem stayed the same with small differences:

Unhandled exception at 0x77913560 in 3d.exe: 0xC0000005: Access violation writing location 0x00000000001e0e84.

As you can see the code given at the end of the error line is different, I want to know and trace the error. Any kind of help will be appreciated. Thanks !

Emre

[SergeyKostrov]

>>First-chance exception at 0x0000000140617977 in 3d.exe: 0xC00000FD: Stack overflow. Here is a statement from the 1st post: >>>>...The "Stack Size" problem has seemed reasonable. I changed reserve size to 200,000,000 and commit size 20,000,000. Let me provide you with some real numbers for Stack Commit and Reserved values used for a Merge Sorting algorithm ( recursive ) to sort very large data sets ( greater than 64MB on a 32-bit platform ): ...-lS:1073741824 -lSc:1073741824... Stack Commit = 1073741824 Stack Reserved = 1073741824 Note: 1073741824 = 1GB Please try to increase Stack Commit and Reserved values: Stack Commit = 268435456 Stack Reserved = 268435456 Also, power of 2 values are recommended and instead of 200,000,000 use 268,435,456 ( 2^28 ).

[Steven_L_Intel1]

Sergey Kostrov wrote:
Please try to use -heap-arrays 1024 ( or 2048, 4096, etc ) Fortran compiler option.

I agree on using -heap-arrays, but don't bother with a number for the option - the number has no useful effect.

[SergeyKostrov]

>>...I agree on using -heap-arrays, but don't bother with a number for the option - the number has no useful effect. Thank you, Steve for the note! I've noticed that something is wrong because when I tried to use 2048 or 4096, etc I didn't see any difference for a Fortran test application I used.

[emreka82]

Sergey Kostrov wrote:

>>>>>Please try to increase Stack Commit and Reserved values:

Stack Commit = 268435456
Stack Reserved = 268435456

Also, power of 2 values are recommended and instead of 200,000,000 use 268,435,456 ( 2^28 ).

Sergey,

I applied your stack size suggestions but the problem is the same:

Unhandled exception at 0x000000013fbb7977 in 3d.exe: 0xC0000005: Access violation writing location 0x000000000014f000.

The output is:

'3d.exe': Loaded 'C:\Windows\System32\ntdll.dll', Cannot find or open the PDB file
'3d.exe': Loaded 'C:\Windows\System32\kernel32.dll', Cannot find or open the PDB file
'3d.exe': Loaded 'C:\Windows\System32\KernelBase.dll', Cannot find or open the PDB file
'3d.exe': Loaded 'C:\Windows\System32\imagehlp.dll', Cannot find or open the PDB file
'3d.exe': Loaded 'C:\Windows\System32\msvcrt.dll', Cannot find or open the PDB file
'3d.exe': Loaded 'C:\Windows\System32\advapi32.dll', Cannot find or open the PDB file
'3d.exe': Loaded 'C:\Windows\System32\sechost.dll', Cannot find or open the PDB file
'3d.exe': Loaded 'C:\Windows\System32\rpcrt4.dll', Cannot find or open the PDB file
First-chance exception at 0x000000013fbb7977 in 3d.exe: 0xC0000005: Access violation writing location 0x000000000014f000.
Unhandled exception at 0x000000013fbb7977 in 3d.exe: 0xC0000005: Access violation writing location 0x000000000014f000.
The program '[6084] 3d.exe: Native' has exited with code -1073741819 (0xc0000005).

[andrew_4619]

Not quite the same the first exception was stack overflow in the earlier dump now it is access violation.

[emreka82]

app4619 wrote:

Not quite the same the first exception was stack overflow in the earlier dump now it is access violation.

Also, I see that when I run the program again, it stopped at the "malloc.c". But after I run the program agin without any change, it again stops at my recursive subroutine as it was in first time. I don't understand why it goes to malloc.c, it happened only one time.

[andrew_4619]

It may be a problem running out of system resources or it may be some other system problem but first you need to identify the point at which your application where things *start* to go wrong. If it were me I would pick a point in the recursive code and set a conditional break in the VS debugger to break (for example) after every 100 hits. By stepping through you will learn the crash is somewhere between 2200 and 2300 hits for example. Run again stopping after 2200 hits on the break point and then step slower say every 5 hits. This way you will find the point in your code where it goes wrong and can step the program line by line. The proability is you exceed an array or at the start of a new level of recursion you run out of some resourse e.g. memory/stack.

[SergeyKostrov]

>>... I see that when I run the program again, it stopped at the "malloc.c"... There are about ~200 code lines for 4 functions in malloc.c source file. So, where exactly the debugger stops ( a line number )? Also, please attach malloc.c because depending on a version of Visual Studio some differences are possible.

[Steven_L_Intel1]

Use the stack frame dropdown to see where malloc was called from in your source (indirectly). It may be several frames up.

Showing us the error messages from the debugger's output pane is not useful. You need to see what your program is doing at the point of the error. That the error moves around when you make changes suggests to me you are corrupting data somewhere.  Inspector XE can help you find this (probably, not guaranteed.)

[SergeyKostrov]

>>...The maximum array size is 7 to 5 but the recursive subroutines and the pointers increase them. To make it clear: >>An 4x3 array with one-level high 1D array (which is pointed by 4x3 array) inside a recursive subroutine which calls 172 times >>itself and in the 173rd call the code breaks. The array elements are float, in Fortran literature real*8... Hi emreka82. Let's try more resolute actions: 1. Set VM values as follows: Min = 24GB and Max = 32GB 2. Set values Stack Commit = 536870912 and Stack Reserved = 536870912, and Heap Commit = 536870912 and Heap Reserved = 536870912 Note: 536870912 = 2^29 / you could use even higher values, like 2^30 3. Verify that application is compiled for 64-bit in Release and Debug configurations 4. Comment All processing in your recursive subroutines and declare a global counter of recursive calls. So, as soon as it reaches 173 ( or so ), call a return in order to "unwind" all resursion calls 5. If it Did Not crash everything is fine with memory and simplefied Fortran codes. If it is Not increase Stack\Heap Commit & Stack\Heap Reserved values and look for bugs ( ask somebody to do a code review ) 6. The goal is as follows: you need to create a simplified and very clean version of your processing ( Do Not worry about results yet! ) and when it Did Not crash start uncommenting codes line by line or by a couple of lines at a time with a verification that codes are working. Does it make sense?

[Bernard]

>>>Can you tell me or show me how can I change this adresses into location to find ...>>>

You need to ru your program under debugger and for resolving heap allocation issues it is recommended to install Application Verifier.Next you need to track in disassembly window where this address  0x00000000000120ff8 is coming from.I suppose that this is some garbage left on the stack frame which is somehow loaded and dereferenced by your code.

[emreka82]

Sergey Kostrov wrote:

>>... I see that when I run the program again, it stopped at the "malloc.c"...

There are about ~200 code lines for 4 functions in malloc.c source file. So, where exactly the debugger stops ( a line number )? Also, please attach malloc.c because depending on a version of Visual Studio some differences are possible.

I wish I can turn back in time and see the line. It happened only "one" time, unrepeatedly. If it happens again, I will surely look at the line.

[emreka82]

Steve Lionel (Intel) wrote:

Use the stack frame dropdown to see where malloc was called from in your source (indirectly). It may be several frames up

I see that, but cannot reach malloc.c using stack frame. It is not there now, maybe it is because of different VM limits and stack sizes. I changed them several times.

[emreka82]

Sergey Kostrov wrote:

>>...The maximum array size is 7 to 5 but the recursive subroutines and the pointers increase them. To make it clear:
>>An 4x3 array with one-level high 1D array (which is pointed by 4x3 array) inside a recursive subroutine which calls 172 times
>>itself and in the 173rd call the code breaks. The array elements are float, in Fortran literature real*8...

Hi emreka82.

Let's try more resolute actions:

1. Set VM values as follows: Min = 24GB and Max = 32GB

2. Set values Stack Commit = 536870912 and Stack Reserved = 536870912, and Heap Commit = 536870912 and Heap Reserved = 536870912
Note: 536870912 = 2^29 / you could use even higher values, like 2^30

3. Verify that application is compiled for 64-bit in Release and Debug configurations

4. Comment All processing in your recursive subroutines and declare a global counter of recursive calls. So, as soon as it reaches 173 ( or so ), call a return in order to "unwind" all resursion calls

5. If it Did Not crash everything is fine with memory and simplefied Fortran codes. If it is Not increase Stack\Heap Commit & Stack\Heap Reserved values and look for bugs ( ask somebody to do a code review )

6. The goal is as follows: you need to create a simplified and very clean version of your processing ( Do Not worry about results yet! ) and when it Did Not crash start uncommenting codes line by line or by a couple of lines at a time with a verification that codes are working.

Does it make sense?

I applied first 3 actions. The thing is the same but at the end of the "call stack" window, "The maximum number of stack frames supported by Visual Studio has been exceeded" is written. Is it because of "out-of-memory" ? I will apply rest of your action suggestions and see what happens. Thanks.

[emreka82]

Sergey Kostrov wrote:

>>....4. Comment All processing in your recursive subroutines and declare a global counter of recursive calls. So, as soon as it reaches 173 ( or so ), call a return in order to "unwind" all resursion calls

5. If it Did Not crash everything is fine with memory and simplefied Fortran codes. If it is Not increase Stack\Heap Commit & Stack\Heap Reserved values and look for bugs ( ask somebody to do a code review )

6. The goal is as follows: you need to create a simplified and very clean version of your processing ( Do Not worry about results yet! ) and when it Did Not crash start uncommenting codes line by line or by a couple of lines at a time with a verification that codes are working.

Does it make sense?

I added print*, <a number> in the recursive subroutine. This time, it stopped in "write.c" . The stopping line (showns as "HERE IT STOPS" comment) is as follows:

 [cpp]

/***
*write.c - write to a file handle
*
* Copyright (c) Microsoft Corporation. All rights reserved.
*
*Purpose:
* defines _write() - write to a file handle
*
*******************************************************************************/

#include <cruntime.h>
#include <oscalls.h>
#include <ctype.h>
#include <io.h>
#include <errno.h>
#include <msdos.h>
#include <mtdll.h>
#include <stdlib.h>
#include <string.h>
#include <internal.h>
#include <setlocal.h>
#include <locale.h>
#include <wchar.h>

#define BUF_SIZE 5*1024 /* size of LF translation buffer */
/* default buffer is 4K, plus extra for LFs */

/***
*int _write(fh, buf, cnt) - write bytes to a file handle
*
*Purpose:
* Writes count bytes from the buffer to the handle specified.
* If the file was opened in text mode, each LF is translated to
* CR-LF. This does not affect the return value. In text
* mode ^Z indicates end of file.
*
* Multi-thread notes:
* (1) _write() - Locks/unlocks file handle
* _write_nolock() - Does NOT lock/unlock file handle
*
*Entry:
* int fh - file handle to write to
* char *buf - buffer to write from
* unsigned int cnt - number of bytes to write
*
*Exit:
* returns number of bytes actually written.
* This may be less than cnt, for example, if out of disk space.
* returns -1 (and set errno) if fails.
*
*Exceptions:
*
*******************************************************************************/

/* define normal version that locks/unlocks, validates fh */
int __cdecl _write (
int fh,
const void *buf,
unsigned cnt
)
{
int r; /* return value */

/* validate handle */
_CHECK_FH_CLEAR_OSSERR_RETURN( fh, EBADF, -1 );
_VALIDATE_CLEAR_OSSERR_RETURN((fh >= 0 && (unsigned)fh < (unsigned)_nhandle), EBADF, -1);
_VALIDATE_CLEAR_OSSERR_RETURN((_osfile(fh) & FOPEN), EBADF, -1);

_lock_fh(fh); /* lock file */

__try {
if ( _osfile(fh) & FOPEN )
r = _write_nolock(fh, buf, cnt); /* write bytes */
else {
errno = EBADF;
_doserrno = 0; /* not o.s. error */
r = -1;
_ASSERTE(("Invalid file descriptor. File possibly closed by a different thread",0));
}
}
__finally {
_unlock_fh(fh); /* unlock file */
}

return r;
}

/* now define version that doesn't lock/unlock, validate fh */
int __cdecl _write_nolock (
int fh,
const void *buf,
unsigned cnt
)
{ ////HERE IT STOPS
int lfcount; /* count of line feeds */
int charcount; /* count of chars written so far */
int written; /* count of chars written on this write */

[/cpp]

After that, I follow the stack frame. The second line is 74 of write.c, other "call stack locations" are:

3d.exe!for__write_output() + 0x58c bytes

3d.exe!for__put_sf() + 0x129c bytes

3d.exe!for__write_seq_lis_xmit()+ 0x2fbb bytes

3d.exe!for__write_seq_lis() + 0x988 bytes

Then it continues as usual, name of my recursive subroutine and its connected subroutines.

[emreka82]

By the  way, as I add these "print*, <a number>" comments, the breaking point changes. So, I am sure that it runs out of memory.

[Bernard]

@emreka82

Why do not you try to use windbg and application verifier in order to automate the error finding task.

[SergeyKostrov]

>>I applied first 3 actions. The thing is the same but at the end of the "call stack" window, "The maximum number of stack >>frames supported by Visual Studio has been exceeded" is written I'll check it for a 64-bit application. Emreka82, Please try to try to follow steps 4, 5 and 6. As I've already mentioned you need to continue with a simplified and clean version of your processing. In Visual Studio check Advanced properties of the Linker and your Target Machine needs to be MachineX64 (/MACHINE:X64), that is for a 64-bit Windows platform.

[emreka82]

iliyapolak wrote:

@emreka82

Why do not you try to use windbg and application verifier in order to automate the error finding task.

I will give a try, it is on my list. Before installing these kind of  programs, I applied the effects of change in Visual Studio, Virtual Memory etc. Once I am done with these changes, I will install both windbg and application verifier. Thanks iliya.

[emreka82]

Sergey Kostrov wrote:

>>I applied first 3 actions. The thing is the same but at the end of the "call stack" window, "The maximum number of stack
>>frames supported by Visual Studio has been exceeded" is written

I'll check it for a 64-bit application.

Emreka82, Please try to try to follow steps 4, 5 and 6. As I've already mentioned you need to continue with a simplified and clean version of your processing.

In Visual Studio check Advanced properties of the Linker and your Target Machine needs to be MachineX64 (/MACHINE:X64), that is for a 64-bit Windows platform.

I cannot change the target machine. It has "MachineX86 (/MACHINE:IX86)" or "not Set" selections. I have 64bit Windows operating system. Why can't it change ??

[SergeyKostrov]

>>...I cannot change the target machine. It has "MachineX86 (/MACHINE:IX86)" or "not Set" selections. I have 64bit Windows operating >>system... It looks like a Root Cause of your problems because MachineX86 (/MACHINE:IX86) is a setting for 32-bit platforms and 32-bit applications can not allocate more then 2GB of memory. >>...Why can't it change?.. Please ptovide exact details for a Visual Studio you use.