topic Big Size of Matrix error in Fortran in Intel® Fortran Compiler

Big Size of Matrix error in Fortran

aafshani — Sun, 27 May 2012 01:30:54 GMT

Hi everybody;
I'm not computer expert, but during working with my thesis which is about civil Eng., got serious problem with my program seemingly about big size of matrices. I'm working with meshes and using Fortran to analyze entire of my mesh including nodes and elements. So, I have some variables, vectors and matrices. The biggest matrix has around 61,000x61,000 arrays. During debugging of my program, following common message and highlighted ERROR are appearing every time:

--------------------Configuration: Ali-0 - Win32 Debug--------------------
Compiling Fortran...
D:\\Thesis\\Fortran programming\\Copy of shin yokohama fortran\\Ali-0.for
D:\\Thesis\\Fortran programming\\Copy of shin yokohama fortran\\Ali-0.for(25) : Error: A common block or variable may not exceed 2147483647 bytes
& nfix(5000),r1(61000),sk(61000,61000)
----------------------------------^
Error executing df.exe.

Ali-0.exe - 1 error(s), 0 warning(s)

I know the reason is beacasue size of my matrix in bytes outnumbers maximum allocatable bytes for each matrix. I have no choice to use this matrix. I've searched the web for resons and solutions. It seems that this error relates to ram and operating system as well.
I have 2 system 1- Window 32 bit 2GB ram, other Windows 64 bit, 6.00 GB.
Could anybody help me resoling this problem!

Big Size of Matrix error in Fortran

mecej4 — Sun, 27 May 2012 02:46:40 GMT

Do a "back of the envelope" calculation. The array sk requires about 15 GB in single precision and twice that in double precision. These requirements are not reasonable for the hardware available to you.

It is probably true that you do not need dense matrices since matrices arising from discretization of differential equations are sparse. However, you will have to rewrite your programs to exploit that sparseness. MKL provides linear algebra routines that are suited to sparse matrix manipulations.

Big Size of Matrix error in Fortran

aafshani — Sun, 27 May 2012 08:57:16 GMT

Thank you so much mecej4 for your comment;
even if I want to use spareness of the above SK(61000, 61000) matrix, it could be densed to a matrix of SK(61000, 30500) which is still needs lots of bytes.
I've heard there is another solution using so called "Gfortran" as well as Math Kernel Library (MKL). I'm completely new to both of them and litteraly have no idea what are they, how can I use them in my case.
Do you know which one is better to stick with, I mean, which one is much straight forward to use in my case?
Can you please explain a little bit more about how can I use MKL, or is there any simple instruction which shows me how to use MKL with fortran??
Thanks

Big Size of Matrix error in Fortran

mecej4 — Sun, 27 May 2012 09:24:59 GMT

Instead of relying on dubious rumours, seek guidance from a knowledgeable source.

> even if I want to use spareness of the above SK(61000, 61000) matrix, it could be densed to a matrix of SK(61000, 30500) which is still needs lots of bytes.

There is no perceptible basis for that statement, and there are many situations where we know that statement not to be true.

> Can you please explain a little bit more about how can I use MKL, or is there any simple instruction which shows me how to use MKL with fortran??

Take courses on numerical mathematics/numerical analysis/computing.

Big Size of Matrix error in Fortran

jimdempseyatthecove — Sun, 27 May 2012 15:47:28 GMT

"common block or variable may not exceed 2147483647 bytes"

If the array is in COMMON or module, make the array allocatable and perform the allocation in an init subroutine you call at the start of the program.

If the variable is in subroutine/function scope use the heap arrays option.

(this size of allocation will only work in x64 app)

Jim Dempsey

Big Size of Matrix error in Fortran

SergeyKostrov — Sun, 27 May 2012 17:15:09 GMT

Quoting allexberg

...The biggest matrix has around 61,000x61,000 arrays...

[SergeyK] In case ofSingle-Precision 61,000x61000 matrix you need

61,000*61,000*4 = 14,884,000,000 = 13.86GB of memory

--------------------Configuration: Ali-0 - Win32 Debug--------------------

[SergeyK] It is impossible to allocate 13.86GB of memory for one application on a32-bit Windows platform.

...
...\Ali-0.for(25) : Error: A common block or variable may not exceed 2147483647 bytes
...
I have 2 system 1- Window 32 bit 2GB ram, other Windows 64 bit, 6.00 GB.

[SergeyK] You need to use a 64-bit platform with a large amount of physical memory, for example 16GB. You couldtry to
increase a Virtual Memory size up to 16GB on the 6GB platform but performance of yourapplicationwill be significantly affected.

Since your matrix is very largeI would recommend to useDouble-Precision data type to improve accuracy
of your calculations. In that case a 64-bit platform with 32GB of physical memoryhas to be used.

Best regards,
Sergey

Big Size of Matrix error in Fortran

aafshani — Tue, 29 May 2012 06:43:46 GMT

Thanks Jim Dempsey and Sergey Kostrov for your helpful comments;

Sergey Kostrov, you told I have to increase virtual memory up to 16 GB. I checked it. Amount of space available for virtual memory of my platform seems to be 521GB. (I checked it from following address:
Control panel\system\ Advanced system setting\advanced\performance setting->advance tab->virtual memory). Then, I set it to 64GB which already was only 6GB.
I was just trying to test new condition with simple small program with allocatable but large array the same previous sk(61000,61000) matrix, I had before. After running this simple program with 64GB vitual memory, I got another warning like:
".exe has triggered a breakpoint ".
This happens exactly at allocation line of my code.
When I decrease the size of my array, everything is okay. The problem is obviously due to the big size of my array again.

During reading in related Intel forums and other sources; there were some suggestion about involving with stack and heap arrays concepts, as well as increase stack size.
According to the Intel posts, default stack size is around 1MB.
If I want to increase it
1- Is there any idea how much do I have to increase stack size for my case (In the case of sk(61000,61000) matrix) ?
2- What is the maximum stack limit if there is, for example for windoms 64 platform?

Big Size of Matrix error in Fortran

TimP — Tue, 29 May 2012 11:22:36 GMT

Win32 default thread stack size is 1MB, but you didn't give any indication why you mention that here. This is distinct from the stack size which you set in /link /stack: or by editbin, which has a much larger default. Intel libraries, such as you get with MKL, boost the thread stack size default to 2MB for win32 and 4MB for X64, and provide function calls and environment variables to control it. All probably irrevelant to the subject you have been discussing.

Big Size of Matrix error in Fortran

anthonyrichards — Tue, 29 May 2012 13:04:53 GMT

I think you need to rethink the size of your arrays, which appear directly related to the 'fineness' of your mesh.
you are clearly asking for way too much than can be catered for on your system, without rolling stuff in and out from disk.

Do you really need the fineness that requires 61000 divisions?

Are the effects you are modelling likely to have a range covering the whole width of your mesh?

If the effects are likely to be confined to a local area, then you should produce local meshes using much fewere points.

If the range of the effects you are modelling reach over the whole physical space spanned by your mesh (i.e. have long 'wavelengths'), then I suggest reducing the mesh fineness, as variations over small regions are likely to be small and could eventually be filled in using interpolation from a coarser mesh.

Big Size of Matrix error in Fortran

jimdempseyatthecove — Tue, 29 May 2012 16:57:06 GMT

Is your build configuration Win32 or x64?
If Win32 then you are building a 32-bit application (which will run on an x64 platform but with the 2/3 GB memory restrictions).

To convert to 64-bit in VS click on the pull-down arrow on the platform (Win32), then New, then you should get or find x64, also choose to import settings from Win32.

Rebuild and run your test program.

Jim Dempsey

Big Size of Matrix error in Fortran

SergeyKostrov — Wed, 30 May 2012 02:25:52 GMT

Quoting allexberg

...After running this simple program with 64GB vitual memory, I got another warning like:
".exe has triggered a breakpoint ".
This happens exactly at allocation line of my code.

[SergeyK] I understood that it still can't allocate a memory block for61,000x61,000xSizeof(your-data-type) matrix.
My question is: Did you try to execute a 32-bit program on the 64-bit platform?

When I decrease the size of my array, everything is okay. The problem is obviously due to the big size of my array again.

[SergeyK] Please, give exact numbers. Is it fora 32-bit application or 64-bit application?

During reading in related Intel forums and other sources; there were some suggestion about involving with stack and heap arrays concepts,
as well as increase stack size.
According to the Intel posts, default stack size is around 1MB.
If I want to increase it
1- Is there any idea how much do I have to increase stack size for my case (In the case of sk(61000,61000) matrix) ?

[SergeyK] Let me answer in as generic as possible way:

it has be greater than 61,000x61,000xSizeof(your-data-type)

2- What is the maximum stack limit if there is, for example for windoms 64 platform?

[SergeyK] I thinka couple of TBs( it could besomething like 0xFFFFFFFFFFFFFFE0 ).

You need tobuild a 64-bit version of your program and I would recommend to use a memory from the heap instead of
from the stack. Please post updates on your progress.

Best regards,
Sergey

Big Size of Matrix error in Fortran

John_Campbell — Wed, 30 May 2012 12:51:20 GMT

My understanding of this problem is if you use ALLOCATE in a 64-bit version, then there should be no need to consider the heap or stack size.
My preference is to declare the array as allocatable inside a module, such as:

Module Big_Array
real*8, allocatable, dimension(:,:) :: sk
end module Big_Array

If the size of array "sk" is significantly larger than the available physical memory, but less than the size of the pagefile.sys, then expect a long wait for the program to run.
You will benefit from many of the old programming techniques that suited paged memoryin the 70's and 80's,as demonstrated by the run time difference between

do i = 1,n
do j = 1,n
sk(i,j) = 0
end do
end do

compared to

do j = 1,n
do i = 1,n
sk(i,j) = 0
end do
end do

With n = 61,000, my estimate is this is a 28 gb array.
You will probably find that any approach that considers either sparsity, symmetry or the banded nature of the array will significantly reduce both the storage demand and the run time. Look for these savings.

John

Big Size of Matrix error in Fortran

jimdempseyatthecove — Wed, 30 May 2012 13:11:23 GMT

For what its worth. Yesterday I changed the properties on my Windows 7 x64 from 16GB to 128GB. Then ran a test program

real(8) :: skd(61000,61000)
real :: sk(61000,61000)
...
allocate(skd(61000,61000), STAT=i)
(success)
do i=1,61000
skd(i,i) = 0.0
end do
(success)
allocate(sk(61000,61000), STAT=i)
(fail)
??? I do not know why this failed, I think the error code was 47, but I am not sure

Due to the above (working part) filling the diagonal of skd, the Virtural memory commit remained under the 16GB. So I changed the above to

do j=1,6100
do i=1,61000
skd(i,j) = 0.0
end do
end do

When page file started expanding, it never quit (after 2 hours runtime). I had to reboot my system.
I have not re-run the test since.

I've felt Windows Virtual Memory system was severely lacking when configured for applications with memory requirements many times larger than physical memory.

BTW I've written two virtual memory operating systems myself, and other than following John's advice of efficiently sequencing your loop controls, you can get relatively good performance out of an application much larger than physical memory.

Jim Dempsey

Big Size of Matrix error in Fortran

SergeyKostrov — Thu, 31 May 2012 03:02:23 GMT

Quoting jimdempseyatthecove

For what its worth. Yesterday I changed the properties on my Windows 7 x64 from 16GB to 128GB. Then ran a test program
...
I do not know why this failed, I think the error code was 47...

Jim,

Do you think it was returned by aGetLastError Win32 API function? I'm interested to see a screenshot, if possible.

By the way, just checked MSDN andfor the GetLastError function the code 47 is not listed at all.

I wonder ifthe code 47 is some internal errorcode ofthe Fortran compiler?

Best regards,
Sergey

Big Size of Matrix error in Fortran

John_Campbell — Thu, 31 May 2012 05:03:59 GMT

Jim,

Like you, I have also felt that Windows Virtual Memory (WVM)is severely lacking.
However, it is very difficult to provide an objective test.
I developed most of my understanding of virtual memory using Prime and Vaxcomputers. I considered Prime's virtual memory to be superior to any other I have used (certainly more flexible than Vax), although the expectation we have on disk response for paging has changed a lot in 20 years.
From what I can remember, the paging space in 1980 was 2 platters of a 300mb CDC drive or about 40 MB, which is a lot less than the 128 GB you are testing today.
My present approach is to never use paging while the program is running and make sure the memory footprint of the program is less than 80% of the physical memory installed.
My "out of core" algorithms that I have to address larger problems are far superior to WVM.

For those few lazy times when you want a quick solution and rely on WVM, it is a bit of a worry to hear it doesn't work. I experienced a similar problem with Win2k when resorting to WVM years ago, when installed memory was much less. (Even then, it wasquicker to develop an out-of-core approach, than solve the problems of windows paging.)

It would be good to know the reason that it has failed this time, as these few bad examples can lead to a view of WVM that may not be valid. Past experience was that identifying thereal error was very elusive, so I resorted to a solution I knew would lead to an answer.

Alex should be looking at a solution that can run in the available physical memory.

John

Big Size of Matrix error in Fortran

aafshani — Thu, 31 May 2012 12:55:57 GMT

I appreciate all you guys;
>Tim: "Win32 default thread stack size is 1MB, but you didn't give any indication why you mention that here. This is distinct from the stack size which you set in /link /stack: or by editbin, which has a much larger default. Intel libraries, such as you get with MKL, boost the thread stack size default to 2MB for win32 and 4MB for X64, and provide function calls and environment variables to control it. All probably irrevelant to the subject you have been discussing."

Thanks Tim for your advice......

>anthonyrichards: "Do you really need the fineness that requires 61000 divisions?
Are the effects you are modelling likely to have a range covering the whole width of your mesh?"

Yeah, actually, that matrix of Skis just one of my matrices, but the biggest. Even if I want to shrink my mesh,still it will be too large to get the errro again, and another issue is that unfortunately it is not my only array. I have similar arrayand matrices in my program.This is 3D mesh of Soil mediumfor tunnelling simulation.

>Jim Dempsey:"Is your build configuration Win32 or x64?

If Win32 then you are building a 32-bit application (which will run on an x64 platform but with the 2/3 GB memory restrictions). To convert to 64-bit in VS ......."

>SergeyK: "Please, give exact numbers. Is it for a 32-bit application or 64-bit application?"

I am using Visual studio 2008. My build ConfigurationI was Win32 (and, at the moment, I was using my windows 64bit system). So, yeah,ckecked it from Project Property\platform\win32. Next step I changed it to 64x from the same window\configuration managment\.. and changed Active solution platformto 64x.

>[SergeyK] "Let me answer in as generic as possible way, it has be greater than 61,000x61,000xSizeof(your-data-type):"

I changed theactive platform to 64X and there was no problem, and mention again that I increase my virtual memory up to near 100 GB. I was able to run the program up to the half way. All of the sudden another error. The error on the console window was:
forrtl: server (157): program exception-access violation
message on the pop up window (Microsoft visual studio) was:

unhandled exception at 0x000000014000da49 in(name of my file.exe): 0xc0000005: Access violation writing location 0x0000000000000004

I checked the program itself, there was no problem about allocation, I think it agan relate to the memory.
Any thought on this??

It is worthy to mention that inmy program, I have many variable, arrays, and matrices, but most of them are small size. The biggest one was the SK matrixI mentioned which I manged to shrink to Sk(55000,8000). But, I think as the program goes on,volume of used memory is increasing up to the point that facing no memory. So, I tried to makemy big size arrays allocatable including Sk. But, still got above message. I'm not sure is this error becauase of memory or an internal error of program!!
Thanks.

Big Size of Matrix error in Fortran

Les_Neilson — Thu, 31 May 2012 13:36:38 GMT

It is likely that you have an array bounds being exceeded.
In Project -> Properties -> Fortran -> Run-time I suggest that you turn on "Generate Traceback Information" and "Check Array and String Bounds".
The traceback will hopefully tell you which routineis failing and the line number, and if it is a bounds error, the array and value of the subscript.
Once you have fixed the problem you can turn the checks off again.
(Personally I prefer to leave traceback on, so if a client should ever get one of these errors I can get more info of where the problem lies. YMMV)

Les

Big Size of Matrix error in Fortran

aafshani — Thu, 31 May 2012 14:02:23 GMT

Neilson thanks,
Both of the options youmentioned were default "on" when I checked them.

Big Size of Matrix error in Fortran

jimdempseyatthecove — Thu, 31 May 2012 17:13:04 GMT

John,

>>My present approach is to never use paging while the program is running and make sure the memory footprint of the program is less than 80% of the physical memory installed

Assume for the moment you have an application data requirement that is several/many times that of physical memory.

Solution 1:

Rewrite code to use file I/O

Solution 2:

Rewrite code to assure array indexes (in do/for loops) are ordered favorably for virtual memory access

Note, Solution-2 may require no changes, had you originally organized your loops in a paging favorable manner (which by the way is also cache favorable w/rt TLB).

When some rewrite for Solution 2 is necessary, it generally requires little effort.

Should access be totally random, then Solution 1 is not a solution either.

Jim Dempsey

Big Size of Matrix error in Fortran

jimdempseyatthecove — Thu, 31 May 2012 17:24:43 GMT

>>Access violation writing location 0x0000000000000004

Writing into first 4KB of memory is generally indicative of a NULL reference/pointer.
This can be due to:

incorrect calling (args do not match between caller/callee)
failing to allocate
calling library (C) functions with incorrect calling parmeters
writing to stack based array,outside of bounds, and trashing a pointer/reference/array descriptor.

In Debug build, you usually can find the location in your source where the error occured (Call Stack). When error occurs in library function then sometimes the call stack does not expose the caller as source+line.

Jim Dempsey