sizeof(x)

[Ilie__Daniel]

Hello!

I want to write a small routine that would do the following:

• accept one argument, which can be anything (scalar or array of different ranks; integer, real or character)
• retrieve how much memory the dummy argument needs
• and update the total momory so far

The ultimate goal would be to see how much memory do all my allocate statements require.

I thought of the following implementation:

module store

integer(8) :: total = 0

contains

subroutine get_mem(x)

implicit none

total = total + sizeof(x)

end subroutine get_mem

end module store

Now my question is what do I declare x as?

I have allocate statements in specific places in the code, so I could just add the line

total = total + sizeof(...)

but this seems somewhat error prone

I would appreciate any suggestions that you may have.

[Steven_L_Intel1]

There is no way to do this. Any way that you could declare "x" so that it accepted "anything" would prevent use of sizeof(). You'll have to do it the "error prone" way.

[Ilie__Daniel]

Ok, Steve.
Thank you.

[netphilou31]

Hi,

To Steve,

Is there any other way? I was thinking on a generic module procedure that calls several typed prodecures (for scalars, arrays, integer or double). It would be a little bit more complicated but must do the job, no?

Regards,

[Steven_L_Intel1]

If you want to be fully generic, you need an awful lot of procedures - potentially an infinite number. You can get by without array entries by defining an elemental procedure that will get called once for each element, but that could get awfully slow. But an elemental can't reference a global, so....

I think that if I were doing this for an application, I'd write a set of allocate routines for the types of objects I use in the program. This would declare the argument of the correct type (and rank), and have separate dimension arguments. The routine would both do the allocate and increment the global total. I'd have deallocate routines too. But you would not be able to capture automatic (re) allocations or deallocations.

[John_Campbell]

Daniel,

Rather than use SIZEOF, you could try to manage the use of SIZE. This works with arrays of any type. You need to check that allocatable arrays are allocated and you must know the number of bytes associated with a single element of the type. For standard types, KIND = bytes in ifort.
I do not know how easilySIZE works with derived types or how you could easily calculate the bytes associated with a single element of a derived type. If you accept that you need to supply this info explicitly, you might be able to get some idea of the memory in use.
There are ways of managing this calculation, it depends on how accurately you need to calculate it.

The other parameter that can be important when using the result of this calculation is how much memory do you have available. Is it :
the virtual memory limit,
the physical memory limit,
the free memory or
some % of physical memory.

I try to target about 80% of the physical memory. You can have this parameter in a configuration file somewhere standard on your PC. I use to store it in C:\windows, but Windows 7 made that difficult to update.

Hope this helps,
John

[jimdempseyatthecove]

Look at the C heap diagnostic routines. In particular _heapwalk

There is a good example in the VS 2010 help.

Construct a function based on this example. Call this at your program start to determine size of allocations at start of program. Then periodically call this at appropriate points in your program to obtain the current in-use at that point in execution of the program.

Note, depending on how the heap is initialized (Low Fragment Heap or not), you may not be able to depend on the highest fragment address (iow start of last node) as an indication of maximum memory requirements. When LFH is off (default), the heap is somewhat like a ring buffer.

Jim Dempsey

[SergeyKostrov]

Quoting Daniel I.

...
The ultimate goal would be to see how much memory do all my allocate statements require...

What about the Windows Task Manager? It allows to see 'Mem Usage' and 'VM Size' for an application without
any programming.

Best regards,
Sergey

[Steven_L_Intel1]

The task manager won't tell you how much memory the application currently has allocated. It will only tell you the highest amount mapped by the process, which may be significantly more. But if all you want is a relative measure of memory use, it will serve as a proxy.

If you're trying to track memory leaks, then the memory analysis features of Intel Inspector XE can be useful.

[Bernard]

@Sergey I would recommend to use Process Explorer.The ultimate tool for heap walking memory leaks and memory consumption is windbg.

[SergeyKostrov]

Quoting iliyapolak

...The ultimate tool...

for memory leaks detection is a Memory Leaks Detection ( MLD )subsystem I've implemented in 2009 for
the ScaLib project. It is only1,051 C/C++ code lines, absolutely portable (Windows platforms:Desktop,Mobile and CE;
Linux,MS-DOS, etc ),reliable and allows to release all "lost" memoryblocks if needed ( similar to Java's Garbage Collector ).

Actually, everybody uses a toolhe/she likes. :)

Best regards,
Sergey

[Bernard]

IIRC Microsoft LeakDiag tool is specificaly created for memory leaks testing.It is free and you can give it try.

[SergeyKostrov]

Hi Daniel,

Quoting Daniel I.

...
The ultimate goal would be to see how much memory do all my allocate statements require
...
I would appreciate any suggestions that you may have...

Here are a couple of suggestions:

- Consider PDH ( Performance Data Helper )Win32 API
( You will need to create Fortran-wrappers for several functions which could provide all statistics you need /
on MSDN: "Platform SDK: Performance Monitoring" and "Using the PDH Interface" topics)

- Take a look at PVIEW.exe utilityand any version of Platform SDK for Windowsshould haveit
( I think the source codes could be found on Microsoft's website )
Please take a look at a screenshot and as you can see it shows lots of statistics.

- Take a look at PERFMON.exe utility and it is includedwith almost allWindows OSs

In any case you will need to spend some time on learning some small subset of Win32 API.

Best regards,
Sergey

[SergeyKostrov]

Quoting Sergey Kostrov

...- Take a look at PVIEW.exe utilityand any version of Platform SDK for Windowsshould haveit
( I think the source codes could be found on Microsoft's website )
Please take a look at a screenshot and as you can see it shows lots of statistics.
...

NOTE: 'VM Counts' statisticsalso could be used.

EDITED:
Values in fields 'Peak PF' and 'Cur PF' ( section 'Pooled Quotas' ) show amount of memory allocated ( with 'malloc' CRT function, for example )

[SergeyKostrov]

Quoting Sergey Kostrov

...
- Take a look at PERFMON.exe utility and it is includedwith almost allWindows OSs
...

You could also get any performancestatistics with 'SysMon.ocx' ActiveX control:

- C:\WINDOWS\System32\sysmon.ocx
- CLSID: C4D2D8E0-D1DD-11CE-940F-008029004347

Let me know if you need a generic example on how to use it.

[SergeyKostrov]

Note: This is NOT related to the subject of the post and this is an answer to Iliya.

Quoting iliyapolak

IIRC Microsoft LeakDiag tool is specificaly created for memory leaks testing.It is free and you can give it try.

It can not be used on Non-Windows OSs.

A Memory Leaks Detection ( MLD ) API is an internal feature of the ScaLib library. It is an integral
part of the development process and debugging. It means, that a software developer doesn't need to do
anything else in order to verify an application for memory leaks and as soon as debugging is completed
there is some report in the 'Output-Debug' window of a Visual Studio:

[ Case 1: Memory Leaks NOT detected ]
...
'ScaLibTestAppD.exe': Loaded 'C:\WINDOWS\system32\comctl32.dll', Exports loaded.
'ScaLibTestAppD.exe': Loaded 'C:\WINDOWS\system32\shell32.dll', Exports loaded.
'ScaLibTestAppD.exe': Loaded 'C:\WINDOWS\system32\imm32.dll', Exports loaded.
...
Memory Blocks Allocated : 7
Memory Blocks Released : 7
Memory Blocks NOT Released: 0
Memory Tracer Integrity Verified - Memory Leaks NOT Detected
Deallocating Memory Tracer Data Table
Completed
The program '[1644] ScaLibTestAppD.exe: Native' has exited with code 0 (0x0).
...

[ Case 2: Memory Leaks detected - Full Report ]
...
'ScaLibTestAppD.exe': Loaded 'C:\WINDOWS\system32\comctl32.dll', Exports loaded.
'ScaLibTestAppD.exe': Loaded 'C:\WINDOWS\system32\shell32.dll', Exports loaded.
'ScaLibTestAppD.exe': Loaded 'C:\WINDOWS\system32\imm32.dll', Exports loaded.
...
* Memory Block: 0 *
c:\workenv\appsworkdev\common\prttests.cpp(37773)
Memory Block State: 4 - Released by Memory Tracer - Warning!
Memory Blocks Allocated : 7
Memory Blocks Released : 7
Memory Blocks NOT Released: 0
Memory Tracer Integrity Verified - Memory Leaks Detected
Deallocating Memory Tracer Data Table
Completed
The program '[816] ScaLibTestAppD.exe: Native' has exited with code 0 (0x0).
...

[Bernard]

@Sergey

Does ScaLib contain or use some proprietary heap-walking algorithm or maybe it hooks WinApi heap allocation/deallocation functions to obtain informations about heap block addresses and flags.?

[jimdempseyatthecove]

Sergey,

Tools like PVIEW tellsyou your Virtual Memory footprint, not the amount of heap used (or maximum amount of heap used). Many of the current heap designs grow virtual memory much larger than minimally requred. IOW, allocates obtain from "never been allocated" first and from the returned nodes second. Allocations tend to be faster since you are not searching through a linked list of returned nodes. It also has a side effect that protects against (somewhat) bad code where a buffer is used after it was returned. The downside is it expands the virtual memory foot print faster/larger than necessary. The allocation routine does have some parameters as to when not to allocate this way (i.e. to search the list of returned nodes).

Jim Dempsey

[SergeyKostrov]

Quoting iliyapolak

@Sergey

Does ScaLib contain or use some proprietary heap-walking algorithm or maybe it hooks WinApi heap
allocation/deallocation functions to obtain informations about heap block addresses and flags.?

Hi Iliya,

No, it "intercepts" all calls to 'new' /'delete' C++ operators and 'malloc' / 'free' CRT functions and keeps records
in anin-memory database. In essence, two counters are verified on exit from an application:

Number of'new' or 'malloc' calls
Number of'delete' or 'free' calls

When everything is right these two numbers are equal( "balanced" ), for example, 100 'new' C++ operators are called and
100 'delete' C++ operators are called as well. If these two numbers are not equal ( "not balanced" ) then a detailed report
could be displayed in a console window, or in Output-Debug window of a Visual Studio ( a double-click from
thereopens aC/C++ sourcefile and shows a linewherethe memory was allocated ).

A requirement for the subsystem was as follows: As Simple As Possible and it Must Work on ALL Platforms.

Best regards,
Sergey

[John_Campbell]

I have a subroutine that approaches this estimation of used memory in a different way.
By repeated calls to a routine which:
- finds the largest available free memory location,
- then allocates this location,
I can calculate the memory in use.

By tracking the start and end memory addresses of allthe located blocks, I can compute the amount of memory that is not free. This can be assumed to be the memory that is in use.
The key functions to perform are :
1) finding the largest free space at each itteration and
2) releasing all memory that has been allocated during this test, so the calling program can proceed.

It is easier to map in 32-bit, but Ihave adapted a 64-bit version.
Interestingly, this reports a large chunk ofun-available memory from memory address 1.1gb to 5.1 gb. (Not sure why).
Also, the upper memory limit I am reportingon my PC is at about 50gb, so I need to confirm this reported limit. (may be result from pagefile.sys configuration or maximun size of a rank_1 vector ?)

By calling this routine at various stages through the runing of the program, I am able to identify how much memory is being used and how much free (virtual) memory I do have available

This approach may give you the answer you are wanting.

John

PS: I have found some unusual results after adapting my 32bit memory scanning routine to 64bit.
Basically I am getting an error trying to allocate an integer*4 array of, say[bash] integer*4, allocatable, dimension(:) :: ii integer*8 m integer*4 iostat ! m = 198359280 allocate (ii(m), stat=iostat) ! ! iostat returned as 41 [/bash]

However, after allocating a smaller array of m = 198,359,279 ( 1 smaller !) (756mb) I am then able to do a new sequence of ALLOCATE with 198,359,280 being successfull and further increasing values of m up to 1,156,831,232, then failing at 2,082,296,192, before finally succeeding at 1,387,293,696 (5292mb).

In win32 I have used a similarprocess of identifying an upper (invalid)and lower bound (valid) for locating the largest free memory space, but in Win64, this maximum identified free space size appears to be a local maximum to the routine call, whichcan later change to a larger available size.

The principle of the program is :
DO pass = 1,20
m =find_largest_free_memory (pass)
allocate (ii_(m))
END DO

where "find_largest_free_memory" tests different values of m, establishing an upper bound failure then bisecting lower and upper until a highest valid size is found.

Why should successive searches for this largest free space be increasing, when the only change during the run is the allocation ofadditional arrays?
What can temporarily cause memory to not be available, or increase the available memory?

The resulting free memory report I obtain is (requires courier text):

[bash] Free Memory Mapping Blk Used Block Free Start Free End Gap Size_b Size_mb time id k bad 0 0 0 0 0 0.000 0.0000 0 0 0 1 7,077,960 7,077,960 800,515,075 793,437,116 756.681 0.3900 2 37 0 2 7,236 800,522,312 823,850,307 23,327,996 22.247 0.6400 4 26 0 3 2,820 823,853,128 836,813,123 12,959,996 12.360 0.7490 5 24 0 4 16,132 836,829,256 849,789,251 12,959,996 12.360 0.8270 6 24 0 5 16,132 849,805,384 873,133,379 23,327,996 22.247 0.8900 7 26 0 6 2,820 873,136,200 886,096,195 12,959,996 12.360 0.9680 8 24 0 7 16,132 886,112,328 909,440,323 23,327,996 22.247 1.0610 9 26 0 8 2,820 909,443,144 916,643,139 7,199,996 6.866 1.1700 10 23 0 9 8,964 916,652,104 939,980,099 23,327,996 22.247 1.3420 11 26 0 10 2,820 939,982,920 952,942,915 12,959,996 12.360 1.4510 12 24 0 11 16,132 952,959,048 976,287,043 23,327,996 22.247 1.5760 13 26 0 12 2,820 976,289,864 983,489,859 7,199,996 6.866 1.6700 14 23 0 13 8,964 983,498,824 1,006,826,819 23,327,996 22.247 1.7790 15 26 0 14 2,820 1,006,829,640 1,019,789,635 12,959,996 12.360 1.8880 16 24 0 15 16,132 1,019,805,768 1,043,133,763 23,327,996 22.247 1.9820 17 26 0 16 2,820 1,043,136,584 1,050,336,579 7,199,996 6.866 2.0750 18 23 0 17 8,964 1,050,345,544 1,073,673,539 23,327,996 22.247 2.1690 19 26 0 18 4,295,692,380 5,369,365,920 5,370,414,495 1,048,576 1.000 0.0000 -1 0 0 19 64,168 5,370,478,664 43,149,023,667 37,778,545,004 36028.430 0.2970 1 50 0 20 9,876 43,149,033,544 48,698,208,327 5,549,174,784 5292.105 0.5930 3 44 0 vacant memory = 44,395,229,416 41.3463 gb memory in use = 4,302,978,912 4.0075 gb memory released = 44,386,980,844 41.3386 gb memory lost = 7,199,996 0.0067 gb [/bash]

ID represents the order, where id=3 is bigger than id=2, as is id=7,9,11,13,15,17, and 19 all testing larger available sizes than the previous call.
I only adapted the program today, changing to integer*8 variables, so there may be an error but the program runs successfully as a Win32 program.

[bash] integer*8 function Get_Free_Memory_Block (nb, test_count) ! ! This routine searches for the largest available memory block still available ! LF95 does not give non-zero STAT if array is > 2gb ! lf95 uses l = pointer (ii) for memory address ! integer*4 nb, test_count ! number of tests ! integer*4, allocatable, dimension(:) :: ii ! integer*8 m_low, m_high, m, l integer*4 iostat, k, k1 real*8, parameter :: one_mb_r4 = 1024. * 1024. / 4. real*8 mb character bus*15, bus_l, bus_low external bus data m_high / -1 / ! m_low = 0 ! if (nb <= 1) m_high = -1 ! test for high point m_high = -1 m = 1000000 ! first size for start 4mb k1 = 0 ! do k = 1, huge(k) ! ! look for new estimate if (m_high < 0) then m = m*1.8 ! step less than x2 for win32 else m = (m_low+m_high) / 2 if (m == m_low) exit end if ! allocate (ii(m), stat=iostat) ! if (nb>0) write (*,9001) 'get_free_mem',k,m,loc(ii),iostat 9001 format (a,i4, 2i14,i6) ! if (iostat == 0) then ! valid test could be higher m_low = m l = loc (ii) deallocate (ii) else ! invalid test, must be less if (m_high < 0) k1 = k ! first fail m_high = m end if ! end do ! mb = m_low ; mb = mb / one_mb_r4 bus_l = bus (l) bus_low = bus (m_low) write (*,2001) nb,' Get_Free_Memory_Block ', bus_l, bus_low, mb, k1, k 2001 format (i3,a,a,a,2x,f10.3,' mb : tests ',2i5) ! test_count = k Get_Free_Memory_Block = m_low ! end function Get_Free_Memory_Block [/bash]

[jimdempseyatthecove]

John,

The problems with your allocation method to determine unused portions of memory are:

1) It forces your Virtual Memory size to maximum permitted size (and may cause page file expansion and/or GP fault due to failure of page file expansion).

2) It does not account for memory used at start of your test.

Jim Dempsey