Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Intel Community
- Software Development SDKs and Libraries
- Intel® oneAPI Math Kernel Library & Intel® Math Kernel Library
- Vector size 10 000 000 memory problem

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

steingre

Beginner

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

02-14-2012
06:41 AM

31 Views

Vector size 10 000 000 memory problem

I create a vector of size 10 million and I need to compute a distribution statistic from the numbers.

Once I call the function I wrote I get the following error message:

Program received signal: EXC_BAD_ACCESS.

sharedlibrary apply-load-rules all

No memory available to program now: unsafe to call malloc

Data Formatters temporarily unavailable, will re-try after a 'continue'. (Cannot access memory at address 0x7fff5afb3618)

Does that mean that fortran cannot handle the vector size? Do you have an idea of how I might be able to solve the problem? However, I do need the 10 million values in the vector to compute my summary statistic.

If I do the same calculations with vector size 100 000 everything works fine.

I appreciate any help!

Best,

steingre

Link Copied

5 Replies

Chao_Y_Intel

Employee

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

02-14-2012
11:38 PM

31 Views

Hello,

This looks a general program problem, other than specific MKL question. When I tested some simple code, it looks that such size of malloc could work well. I just test with the following code:

program test_malloc

integer i

real*8 x(*)

pointer(ptr_x,x)

i=80000000

ptr_x = malloc(i)

print *, ptr_x

end program test_malloc

In your code, does it have other large memory allocation, which already uses most of the memory space? Also is it a 64 bit application, or 32 bit?

Thanks,

Chao

Andrey_N_Intel

Employee

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

02-15-2012
12:17 AM

31 Views

Hello Steingre,

In addition to answers to Chao's questions if you could provide additional details, it would help us tobetter understand the issue you communicate and ways for its resolution:

1. your environment: OS, Fortran (compiler) version,CPU,its mode, IA-32 or Intel 64, Intel MKL version

2. compiler switches you use to build the application

3. short description of the computations you do (for example, compute mean and variance for a dataset of 1 million double precision numbers). If you could provide a short test case that would be even perfect.

Thanks,

Andrey

steingre

Beginner

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

02-15-2012
05:19 AM

31 Views

I use the Intel Fortran 11 compiler with the MKL 10.3.

I run the x86_64 bit architecture on MAC OS X 10.6.

The processor is 2.13 GHZ Intel Core 2 Duo and memory 4GB 1067 MHZ

I compute the Theil index, which is a concentration measure similar to the Gini coefficient.

I wrote a Theil function to compute the concentration index of the rev variable, which is a vector with sizeNNUm = 1 000 000 (in the ideal case higher, i.e. 10 000 000)

!============================================================================

function Theil(rev,NNum) result(T)

integer, intent(in) :: NNum

real rev(NNum)

real ts(NNum), AS(NNum)

real T, mean, nnul

AS=0

ts=0

nnul=0

do i=1,NNum

AS(i)=rev(i)

if (AS(i).gt.0) then

nnul=nnul+1 ! Count number of products

end if

end do

! Compute the mean

mean = sum(AS)/(max(1,nnul))

! Initialize Theil index

T=0

ts=0

! Calcuate the summation values

do k=1,NNum

if (AS(k).ne.0)then

ts(k) = (AS(k)/mean)*(log(AS(k)/mean))

end if

end do

T=sum(ts)/(max(1,nnul))

end function Theil

!============================================================================

Andrey_N_Intel

Employee

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

02-15-2012
06:58 AM

31 Views

As Chao mentioned thisissue looks like general and is not related to the Intel Math Kernel Library.

There are several visible options to resolve issue of big arrays in your application:

1.The first, malloc basedoption is suggested earlier by Chao

2. You also might want to try compiler switch -heap-arrays n whichhelps to allocate arrays of given minimum size n in kilobytes in heap memory, not on stack.

3. Anotheroption is block-based modification of the algorithm for computation of Theil index. It is based on progressive processing of your dataset: instead of keeping huge arrays in memoryyou might have an array or "window" of the fixed size that well fits to memory (and caches) of your server. This windowmoves over your dataset and allows you to accumulate standartization coefficient/mean estimate/logarithms into temporary arrays of fixed size and variables. Using this option, you would eliminate the need to allocate big arrays in the code.

4.Modification of the option #3which can be based on functionality of Intel Math Kernel Library, in particular, Vectorlogarithmic function and SummaryStatistics algorithm for computation of mean, which might help you to further improve the speed of your application, in addition to vectorization capabilities of Intel Fortran Compiler. Did you try Intel MKL in your code?

Please, letus know if this answers your question.

Also, please, let me know if you need additional details behind the options above.

Thanks,

Andrey

SKost

Valued Contributor II

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report Inappropriate Content

02-15-2012
07:40 AM

31 Views

Quoting steingre

function and it allows to allocate lots of memory in a single call.

[SergeyK]

A '

from '

...

/* Maximum heap request the heap manager will attempt */

#ifdef _WIN64

#define _HEAP_MAXREQ 0xFFFFFFFFFFFFFFE0

#else

#define _HEAP_MAXREQ 0xFFFFFFE0

#endif

...

As you can see even on a **32-bit** system it can allocate big chunks of memory. But, in areal life situation is

very different.For example, on a **32-bit** system it couldn't allocate more than **1.09GB** in a single call. That

is really strange and it puzzles me for a long time!

In your case a**76MB** of memory blockis well below **1.09GB** limit ( a vector of size of ~146,000,000 double-precision elements ).

Best regards,

Sergey

For more complete information about compiler optimizations, see our Optimization Notice.