Vector size 10 000 000 memory problem

[steingre]

Hi,

I create a vector of size 10 million and I need to compute a distribution statistic from the numbers.

Once I call the function I wrote I get the following error message:

Program received signal: EXC_BAD_ACCESS.

sharedlibrary apply-load-rules all

No memory available to program now: unsafe to call malloc

Data Formatters temporarily unavailable, will re-try after a 'continue'. (Cannot access memory at address 0x7fff5afb3618)

Does that mean that fortran cannot handle the vector size? Do you have an idea of how I might be able to solve the problem? However, I do need the 10 million values in the vector to compute my summary statistic.

If I do the same calculations with vector size 100 000 everything works fine.

I appreciate any help!

Best,

steingre

[Chao_Y_Intel]

Hello,

This looks a general program problem, other than specific MKL question. When I tested some simple code, it looks that such size of malloc could work well. I just test with the following code:

program test_malloc
integer i
real*8 x(*)
pointer(ptr_x,x)
i=80000000
ptr_x = malloc(i)
print *, ptr_x
end program test_malloc

In your code, does it have other large memory allocation, which already uses most of the memory space? Also is it a 64 bit application, or 32 bit?

Thanks,
Chao

[Andrey_N_Intel]

Hello Steingre,

In addition to answers to Chao's questions if you could provide additional details, it would help us tobetter understand the issue you communicate and ways for its resolution:

1. your environment: OS, Fortran (compiler) version,CPU,its mode, IA-32 or Intel 64, Intel MKL version

2. compiler switches you use to build the application

3. short description of the computations you do (for example, compute mean and variance for a dataset of 1 million double precision numbers). If you could provide a short test case that would be even perfect.

Thanks,
Andrey

[steingre]

Thank you very much for your comments.

I use the Intel Fortran 11 compiler with the MKL 10.3.

I run the x86_64 bit architecture on MAC OS X 10.6.

The processor is 2.13 GHZ Intel Core 2 Duo and memory 4GB 1067 MHZ

I compute the Theil index, which is a concentration measure similar to the Gini coefficient.

I wrote a Theil function to compute the concentration index of the rev variable, which is a vector with sizeNNUm = 1 000 000 (in the ideal case higher, i.e. 10 000 000)

!============================================================================

function Theil(rev,NNum) result(T)

integer, intent(in) :: NNum

real rev(NNum)

real ts(NNum), AS(NNum)

real T, mean, nnul

AS=0

ts=0

nnul=0

do i=1,NNum

AS(i)=rev(i)

if (AS(i).gt.0) then

nnul=nnul+1 ! Count number of products

end if

end do

! Compute the mean

mean = sum(AS)/(max(1,nnul))

! Initialize Theil index

T=0

ts=0

! Calcuate the summation values

do k=1,NNum

if (AS(k).ne.0)then

ts(k) = (AS(k)/mean)*(log(AS(k)/mean))

end if

end do

T=sum(ts)/(max(1,nnul))

end function Theil

!============================================================================

[Andrey_N_Intel]

Hello,

As Chao mentioned thisissue looks like general and is not related to the Intel Math Kernel Library.
There are several visible options to resolve issue of big arrays in your application:
1.The first, malloc basedoption is suggested earlier by Chao

2. You also might want to try compiler switch -heap-arrays n whichhelps to allocate arrays of given minimum size n in kilobytes in heap memory, not on stack.

3. Anotheroption is block-based modification of the algorithm for computation of Theil index. It is based on progressive processing of your dataset: instead of keeping huge arrays in memoryyou might have an array or "window" of the fixed size that well fits to memory (and caches) of your server. This windowmoves over your dataset and allows you to accumulate standartization coefficient/mean estimate/logarithms into temporary arrays of fixed size and variables. Using this option, you would eliminate the need to allocate big arrays in the code.

4.Modification of the option #3which can be based on functionality of Intel Math Kernel Library, in particular, Vectorlogarithmic function and SummaryStatistics algorithm for computation of mean, which might help you to further improve the speed of your application, in addition to vectorization capabilities of Intel Fortran Compiler. Did you try Intel MKL in your code?

Please, letus know if this answers your question.
Also, please, let me know if you need additional details behind the options above.

Thanks,
Andrey

[SergeyKostrov]

Quoting steingre

I create a vector of size 10 million and I need to compute a distribution statistic from the numbers.

[SergeyK] It isonly~76MB! How much memory do you have on a system and how big is a Virtual Memory file?

Does that mean that fortran cannot handle the vector size?

[SergeyK] It should work.On Windows platforms'malloc' actuallyuses 'HeapAlloc' Win32 API
function and it allows to allocate lots of memory in a single call.

Do you have an idea of how I might be able to solve the problem?If I do the same calculations with vector size 100 000 everything works fine.

[SergeyK]

A 'malloc' CRT-function allocates memory from the heap. Here are some technical data on some limitations
from 'malloc.h' header:

...
/* Maximum heap request the heap manager will attempt */

#ifdef _WIN64
#define _HEAP_MAXREQ 0xFFFFFFFFFFFFFFE0
#else
#define _HEAP_MAXREQ 0xFFFFFFE0
#endif
...

As you can see even on a 32-bit system it can allocate big chunks of memory. But, in areal life situation is
very different.For example, on a 32-bit system it couldn't allocate more than 1.09GB in a single call. That
is really strange and it puzzles me for a long time!

In your case a76MB of memory blockis well below 1.09GB limit ( a vector of size of ~146,000,000 double-precision elements ).

Best regards,
Sergey