How to solve the problem of 'insufficient virtual memory'

[Zhanghong_T_]

In CVF 6.6, when I allocate a large array, program can't execute and displays 'insufficient virtual memory'. I have setthe maximum virtual memory to 4096M from Control Panel. Can I do any other thing to avoid this?

Thanks,

Zhanghong Tang

[Steven_L_Intel1]

4096M is 4GB, which is twice the maximum virtual address space available under Windows. How large an array are you allocating?

[tonyjay68]

How much are you trying to allocate in a single block and in total.

I believe.

1. The most memory any process can allocate in total is 2 GB (except if you have enabled the 3GB switch which I'm sure you haven't or you have windows terminal server which I think is 3gb)

2. The largest single block of memory you can alloacate is bound to be less than 2GB (perhaps 1GB). if you have virus checkers, mouse dlls etc coming in they will fragment your memory.

3. If you have 4GB virtual memory you are probably running into 1 or 2

Tony

[Steven_L_Intel1]

We've seen 1.75GB as a typical maximum allocation size.

[Zhanghong_T_]

In fact, I found my program only used several hundred Megabits before I allocate3 arrays. The arrays I allocated are double precision with size of about 3 million. Is it the reason of 'insufficient virtual memory'? Or I should changesome settings of my project?

Thanks,

Zhanghong Tang

[tonyjay68]

It may help if you show how you are 'allocating memory' and where exactly the error is coming from (what statement and how is the program showing the error i.e. a number of a message).

I assume you are 'allocating' on the heap rather than trying to create arrays on the stack. A code snippet, or a very simple and small example program would help.

Tony Jay

[Zhanghong_T_]

Thank you very much!

In fact, the message displayed when I allocate a large array (allocate(array(largesize))), here largesize is as large as tens of billions and array is a double precision array. I have also got to know the array is allocated on heap rather than on virtual memory. How can I make the array be allocated on virtual memory?

Sincerely,

Zhanghong Tang

[greldak]

Do you really need this big an array? As this will be paging to disk repeatedly would it not be better to rewrite your code to use a file.

If needs be you could then read a portion of it into a smaller and more managable array and carry out whatever you need to do on that - effectively you would be doing your own disk paging.

[Zhanghong_T_]

In fact, I am trying to solve a large sparse matrix whoseorder isup to several million. Now it is difficult for me to form so large matrix (sparse matrix store format).

[Jugoslav_Dujic]

I don't get it -- how can a "sparse matrix store format" take fewGBof memory? The basic idea behind sparse matrix store format is to minimize the storage space.

Which mathematical problem exactly are you trying to solve? You can find a lot of tested mathematical codesusing sparse storage on www.netlib.orgfor example, or http://www-users.cs.umn.edu/~saad/software/SPARSKIT/sparskit.html.

Jugoslav

[Zhanghong_T_]

I have watched the useage of memory of my program. Before allocate arrays, total memory used is 1032M (from Windows Task Manger). Then I allocate three arrays:

a(nz), irow(nz), jcol(nz)

a is double precision;

i and j are integer;

here nz=51,053,158

after run the allocate code, the error message displayed.

But I think it would only allocate less 200M memmory, why display 'insufficient virtual memory'?

[Jugoslav_Dujic]

Because you're already consuming 1GB, other processes have their share as well. You're not allocating 200 MB, but (8 + 4 + 4) bytes *51 million = 816 MB.

Your sparse matrix has 51 million non-zero elements??? What kind of problem is that???

Jugoslav

[Zhanghong_T_]

Sorry, it's my error. But even I allocate 816M, it will not display the error message.

Look at the following example:

Code:

integer, parameter:: maxsize=1000
integer, parameter:: memnum=200

real*8, allocatable:: array(:, :, :)

allocate(array(maxsize, maxsize, memnum))
end

Before running this program, the used memory is about 1030M. After running this program, the used memory is about2,556M. The size of the array is 8*200 million!

[Zhanghong_T_]

Another question:

Before running program, the used memory is about 520M. Then I allocate memory by two different ways:

1. allocate an array a time:

Code:

integer, parameter:: maxsize=1000
integer, parameter:: memnum=250
real*8, allocatable::array1(:, :, :)
allocate(array1(maxsize, maxsize, memnum))
end

The error message displays.

2. Allocate memory many times:

Code:

integer, parameter:: maxsize=1000
integer, parameter:: memnum=250
type mem
  real*8, allocatable:: array(:, :)
  type(mem), pointer:: next=>null()
end type
type(mem), pointer:: memhead, memcurr, newmem
allocate(memhead)
memcurr=>memhead
write(*,*)sizeof(memhead)
do i=1, memnum
  allocate(memcurr%array(maxsize, maxsize))
  allocate(newmem)
  memcurr%next=>newmem
  memcurr=>newmem
  write(*,*)i
enddo
end

This time no error message displayed. Why?

[tonyjay68]

I think you are wrong Jugoslav. We have2GB per process. YOu can run two processes both allocating (say) 1.5 GB and run them onthe same machine (if you have (say) 4 GB of virtual memory.

We get the problem that large arrays can not be allocated because the memory if fragmented (3rd party DLL's etc which have been built at in inappropriate address) we have to rebase them in order to get 1.4 GB of continuous memory, but you can not rebase users 3rd party DLL's ! . If you enable the 3GB switch in the boot.ini, you are almost guaranteed to get 2 *1 GB blocks as well as other little bits

Tony

---

[Zhanghong_T_]

Thank you very much Tony!

If you enable the 3GB switch in the boot.ini, you are almost guaranteed to get

~~~~~~~~~~~~~~~~~~~~~~~~~~~

How can I set this?

2 *1 GB blocks as well as other little bits

We get the problem that large arrays can not be allocated because the memory if fragmented (3rd party DLL's etc which have been built at in

~~~~~~~~~~~~

Thereis only one project and I havenot called 3rd partyDLL

inappropriate address)

Zhanghong Tang

[Jugoslav_Dujic]

My bad. You're right Tony, and you probably hit the nail with the contiguity issues.

I've overlooked the fact that virtual memory can be discontiguous. Physical memory (including swap space)may be totally discontiguous but still mapped into process's virtual address space as contiguous. However, the problem indeed seems to be that dll's can be mapped somewhere right in the middle of contiguous virtual address space, cutting it into half. I'm just looking at Debug->Modules of a large process -- DFORRT.dll and the .exe are low, system dll's are at the very top, but few dll's are somewhere in the middle (0x10000000 - 0x1F000000). The available space in-between is around 1.2 GBin the upper part and bits&pieces below.

Jugoslav

[tonyjay68]

Zhanghong Tang

I think there is a better way to analyse you prolems. Do not use 'computer memory used'

Remembering that your limiting factor is memory per process: Use task manager to look at the process as it allocates (put a user interaction in before each allocation). Make sure you have the 'VM size' column turned on and watch this one NOT memory used.

Also download process explorer (free) from http://www.sysinternals.com/

to see how much continuous memory you have of your running process . (its quite easy to understand the process before its starts allocating, you have to then 'guess' how the memory is being allocated).

If you report the results of this interrogation I may be able to look further.

We regularly run huge probelms with sparse arrays on machines with 2GB of on board RAM, so we do not get any paging. We suffer from 'growing' our arrays (i.e. deallocating and reallocating larger) as if this happens with more than one array you can leave 'holes' in the memory which can not be compressed. The only real solution to this is to rewritecode in a more object based way rather than allocating large arrays , or to move to 64 bit !).

I don't think this is your problem however !, but once you have resent your numbers I'll have a better idea.

Tony

ps one useful routine to write is 'how much is the largest block I can allocate if I wanted to'. Just keep allocating more memory (double for example) until allocation fails. Make sure you deallocated after each success. We have been very suprised when you run this as the first line of you program - some machines comes back at less than 1GB of continous memory due to the DLL importing problem - its very machine specific.

[tonyjay68]

Zhanghong Tang

1. 3rd Party Dll means virus checker, mouse driver, DFORT.dll etc aswell as ones you choseto connect to - you ARE calling 3rd party dlls !.

if this is a problem, put all the DLL causing problems into the same directory as the executable (they will then be used instead of the system ones) and do

something like

Rebase -b 0x70000000 -d -v *.dll

to rebase all the DLLs to the top of memory (i add this as a custom step to my project)

2. the 3GB switch is added to the boot.ini (google to find out how i.e. http://www.sysinternals.com/ntw2k/info/bootini.shtml or http://www.amiravis.com/3GBswitch-instruction.htmlor look in XP help)

BeVERY VERY CAREFULL if you are runing XP SP1, your machine will not reboot at all unless you apply a MS patch http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B328269

The patch must be applied before the boot.ini change (if applicable).

If you do not have much real memory your program will page like crazy with these large arrays , but if you do you load of memory, you will be fine.

Tony

[Zhanghong_T_]

Hi Dr. Tony,

Thank you again!

In fact, I have posted example codes for comparing two ways to allocate memory: ONE time and several times. The former failed and the latter successed.

Also, I have watched the progress of memory allocated from task manager.

Do not use 'computer memory used'

~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sorry, I haven't understood what you said because of my poor English

Make sure you have the 'VM size' column turned on and watch this one NOT

~~~~~~~~~~~~~~~~~~~~~~~~~~

Would you please how can I turn this on?

memory used.

I don't think this is your problem however !, but once you have resent your numbers I'll have a better idea.

~~~~~~

What number?

We have been very suprised when you run this as the first line of you

program

~~~~~~

Which program? The examples I posted before?

Zhanghong Tang

[Zhanghong_T_]

if this is a problem, put all the DLL causing problems into the same directory as the executable (they will then be used instead of the system ones) and do

something like

Rebase -b 0x70000000 -d -v *.dll

to rebase all the DLLs to the top of memory (i add this as a custom step to my project)

2. the 3GB switch is added to the boot.ini (google to find out how i.e. http://www.sysinternals.com/ntw2k/info/bootini.shtml or http://www.amiravis.com/3GBswitch-instruction.htmlor look in XP help)

BeVERY VERY CAREFULL if you are runing XP SP1, your machine will not reboot at all unless you apply a MS patch http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B328269

The patch must be applied before the boot.ini change (if applicable).

I don't dare to do these operations. I am not a computer specialist:(

It seems I have to improve my algorithm, or upgrade my hardware (64 bit). But is there any change when I use a 64 bit PC? OS? Compiler?