- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I need to know what tools to use to determine how much memory my application uses when it is running the largest data files our company has. I would appreciate any help with this. I do not have much experience with the Intel tools.
Thanks.
Michael
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you are on Linux, consider using top
https://www.linux.com/tutorials/5-commands-checking-memory-usage-linux/
If you are on Windows, consider using the Windows Task Manager, or Performance Monitor
https://www.groovypost.com/howto/monitor-pc-memory-performance-usage/
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Jim,
What I really want to find out is how close I am getting to the limits allowed for program memory. I need to determine if I am going to have a stack or heap that is too large. I am trying to determine if I have to change to allocable arrays. The older programmers in the group are concerned that allocating memory will slow down execution.
Michael
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't know of any way to monitor stack usage. There's really no such thing as too large a heap, at least on 64-bit, though total available virtual memory is dependent on swapfile size.
If your program doesn't use allocatable variables now, the limits are fixed when the executable is linked. Some of an effect of switching to allocatable will be how often you do allocate/deallocate. Another, harder to quantify, is that changing from static to heap data means use of descriptors and indirect addressing, which might be noticeable in some cases.
Really, what you want to do is build and run it both ways, profiling with Intel Vtune Amplifier, and see what the total time is as well as where the "hotspots" are.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Steve,
I found an article of yours that indicated that the program and data size in 64 bit windows was the same as 32 unless you allocated arrays. I seemed to remember reading that the issue was a windows file limit.
MIchael
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Stack size is controlled by a linker option (do not go overboard on this setting, especially if you are also using multi-threading)
Heap size is platform (Win32/x64) as well as maximum page file size (which can be larger than physical RAM).
Your program/static data + stack + heap (used) + system space represents your Virtual Memory.
The page file size must accomodate all processes (program/static data + stack + heap (used)) + system space
On Windows, the Linker Object File Format cannot have a segment size larger than 2GB (even on x64). This restricts the total size of static data (segment) to 2GB (as well as total size of Code segment to 2GB). When this limit is approached, then consider making your largest arrays allocatable. The heap (and stack) are not contained within the object file (though size information may be).
You can direct the Linker to produce an object map file, end examine the map file. In top of map file you will see something like this:
Start Length Name Class 0001:00000000 00b18250H .text CODE 0001:00b18250 0000205aH .text$mn CODE 0001:00b1a2b0 00001333H .text$x CODE 0001:00b1b5f0 00000033H .text$yc CODE 0001:00b1b630 00000027H .text$yd CODE 0002:00000000 00000760H .idata$5 DATA 0002:00000760 00000008H .CRT$XCA DATA 0002:00000768 00000008H .CRT$XCAA DATA 0002:00000770 00000008H .CRT$XCF DATA 0002:00000778 00000010H .CRT$XCU DATA 0002:00000788 00000008H .CRT$XCZ DATA 0002:00000790 00000008H .CRT$XIA DATA 0002:00000798 00000008H .CRT$XIAA DATA 0002:000007a0 00000008H .CRT$XIC DATA 0002:000007a8 00000008H .CRT$XIY DATA 0002:000007b0 00000008H .CRT$XIZ DATA 0002:000007c0 0053b510H .rdata DATA 0002:0053bcd0 00000058H .rdata$debug DATA 0002:0053bd28 00000008H .rtc$IAA DATA 0002:0053bd30 00000040H .rtc$IMZ DATA 0002:0053bd70 00000008H .rtc$IZZ DATA 0002:0053bd78 00000008H .rtc$TAA DATA 0002:0053bd80 00000040H .rtc$TMZ DATA 0002:0053bdc0 00000008H .rtc$TZZ DATA 0002:0053bdc8 00018168H .xdata DATA 0002:00553f30 00000000H .edata DATA 0002:00553f30 000000a0H .idata$2 DATA 0002:00553fd0 00000014H .idata$3 DATA 0002:00553fe8 00000760H .idata$4 DATA 0002:00554748 00000dd2H .idata$6 DATA 0003:00000000 000a4f78H .data DATA 0003:000a4f80 02ea4448H .bss DATA 0004:00000000 0000f348H .pdata DATA 0005:00000000 0002d498H .trace DATA 0006:00000000 000009a0H .data1 DATA 0007:00000000 00001200H _RDATA DATA 0008:00000000 000000d0H .rsrc$01 DATA 0008:000000d0 00001448H .rsrc$02 DATA
Those are the named segments. None of them can have a length that exceeds (hex) 7FFFFFFF (7FFFFFFFH) in size.
** and potentially the sum of the start segment number (1st 4 digits before the :) cannot exceed 2GB (7FFFFFFFH)
Add the last entry for a given segment number to the Length (in hex). Open the Windows Calculator, select Programmer view, radio to Hex, enter the last entry's Start for given segment number and then add the Length to get the total size. Then verify this does not exceed 7FFFFFFF.
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In the above, you can see that the largest segment number is 0001
Hex calculation: b1b630 + 00000027 = B1B657 (decimal 11,646,551 which is significantly less than 2GB) This was for the code class. The data class resides in 7 additional segments, the largest of which is 0002 at size 554748 + dd2 = 55551A (decimal 5,592,346).
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I set the linker to produce a map. What folder does it go in. I cannot find it on my machine.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
By default, it goes into your output folder
In the above, it was:
C:\...\x64\Debug
As I was producing an x64 Debug build. If you are producing a Release build on x64 Platform it would be
C:\...\x64\Release
If you are targeting Win32 Platform then remove the "\x64"
(... would be your development folder)
Jim Dempsey
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
My article talked about three different kinds of memory. Code and static data is "static memory", and is indeed limited to 2GB on Windows, though practically it is somewhat less than that. Even on x64. Static data is COMMON, module variables, main program variables, anything marked SAVE or initialized. Procedure local variables can also be static if the procedure is not RECURSIVE (note that Fortran 2018 makes procedures recursive by default, though I am pretty sure Intel Fortran doesn't do that yet.
Stack memory on Windows is allocated by the linker, and comes out of that 2GB static space, which is why it doesn't pay to set the stack size too large as you run the risk of Windows refusing to run your program at all. In Visual Studio this is Linker > System > Stack Reserve Size, on the command line it's the linker option /stack. The default is 10MB - if you find yourself with stack overflow errors, keep doubling that until the errors go away.
Heap memory is that dynamically allocated stuff. On 32-bit it also comes out of that 2GB, but on 64-bit the limit is so large you're unlikely to ever hit it.
One thing that can do you in is array temps - temporary arrays the compiler creates to hold expression results. By default these go on the stack, and if your array is large, you can blow the stack. This is why I recommend using the /heap-arrays (Fortran > Optimize > Heap Arrays > 0) option, which puts these on the heap. Yes, it is more expensive to do this, but the alternative is usually worse - the program crashes. That size option to /heap-arrays is useless - just put zero there.
See also http://software.intel.com/en-us/forums/topic/275071#comment-1548447
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page