Memory leak ( fixed in Update1)

[Mariano_M_]

Edit:  see post #12

Hi all.

My company is considering migrating to IVF from Compaq Compiler.

I'm now testing the trial version of Intel Parallel Studio XE and I've run into a problem with the memory consumption of one of our programs.

The program is an old Fortran code, with static arrais, so the memory is set at the begining of the execution and it should not grow during the run.

This is the behaviour we have observed in several compilers and different OS's ( DEC-alpha, HP.UX, Lahey or Compaq compilers under Windows).

the fact is that when compiled with IVF the system available memory decreases uniformly at a rate of 3MB/s, leading eventually to use all available memory.

Debugging the code I've identified that the "paged pool" (in Process Explorer) raises in blocks of 4KB, and I managed to associate one of these steps to the following write sentence:

     real*8 alf(1)

     character wrd*8

....

     write (wrd,787)alf(i)

787 format(a8)

alf is EQUIVALENCE'd to a big static array and wrd obtains the expected value in that sentence.

I've tried many diffenrent compiler options but no success.

Does anybody know how to solve this issue?

Thanks in advance

M.Martin

[FortranFan]

Are you using the latest Intel Fortran compiler, 16.0?  If not, can you retry using this version?  I could be wrong, but I vaguely recall some memory leaks with READ and WRITE in earlier versions of Intel Fortran that may have been fixed in the latest - may be worth a check.

Separately, as a workaround: are you interested in getting the bit patterns of alf(1) into wrd?  If so, have you considered using TRANSFER intrinsic function: https://software.intel.com/en-us/node/581298 and see if that works.

See this thread for some discussion regarding TRANSFER: https://software.intel.com/it-it/comment/1815943 ;

[Mariano_M_]

Yes, I'm using the latest version, it seems that the leaks have not been fixed yet.

Meanwhile I found a workaround using an equivalence between a real*8 and wrd and using the equivalenced real*8 in comparisons with alf(i).

With this workaround memory leaks dissapear in a simple case, but there are more than 200 READ or WRITE sentences like that in the code, so I'll use your suggestion with TRANSFER, which will be much easier.

Thanks a lot, you saved my week.!

Mariano

[mecej4]

What do you do with the variable wrd after you have written to it with the WRITE statement? Why does wrd have to be of type CHARACTER?

If you make your intentions clear, it may be possible to suggest an alternative to incurring the expense of an I/O call just to transfer eight bytes to a temporary storage location.

[IanH]

FortranFan wrote:

Are you using the latest Intel Fortran compiler, 16.0?  If not, can you retry using this version?  I could be wrong, but I vaguely recall some memory leaks with READ and WRITE in earlier versions of Intel Fortran that may have been fixed in the latest - may be worth a check.

The symptoms might also be a consequence of the handle leak bug that has been encountered with internal IO and ifort 16.0.

[Mariano_M_]

Thanks all for your comments, I've solved the issue with FortranFan's suggestion.

Anyway I think it's an important bug, the program should not eat up memory due to a legal Fortran command, though unnefficient as mecej4 pointed

Regards

[Kevin_D_Intel]

We'd greatly appreciate if you were able to provide a complete small reproducer. With that I can report this to Development who can investigate and identify if this is a known issue (or not) and address any underlying defect.

[Mariano_M_]

Kevin Davis (Intel) wrote:

We'd greatly appreciate if you were able to provide a complete small reproducer. With that I can report this to Development who can investigate and identify if this is a known issue (or not) and address any underlying defect.

Here it goes:

! Console2.f90

!

! FUNCTIONS:

! Console2 - Entry point of console application.

!

!****************************************************************************

!

! PROGRAM: Console2

!

! PURPOSE: Entry point for the console application.

!

!****************************************************************************

program Console2

implicit none

real*8 a,alf

character wrd*8, wrdi*8

integer i

! Variables

! Body of Console2

print *, 'Hello World'

wrdi='anything'

　

read(wrdi,787)alf

print*, wrdi

do i=1,10000000

write (wrd,787)alf

enddo

787 format(a8)

end program Console2

With 10000000 times the program eats about 1.5 GB of RAM, both in Release and Debug mode.

Regards

[NotThatItMatters]

That is the memory leak.  Reference the post "Replacing the first record in a sequential formatted file".

[Kevin_D_Intel]

Thank you for the simple test case. I tried it and concur w/NotThatItMatters and included your case in the internal tracking id, DPD200375683, filed for the earlier cited case, https://software.intel.com/en-us/forums/intel-visual-fortran-compiler-for-windows/topic/593961.

This defect is designated as a must fix for the coming PSXE 2016 Update 1.

(Resolution Update on 11/26/2015): This defect is fixed in the Intel® Parallel Studio XE 2016 Update 1 Release (PSXE 2016.1.051 / CnL 2016.1.146 - Windows)

[Kevin_D_Intel]

I confirmed Mariano's test case in post #8 is fixed by the Intel® Parallel Studio XE 2016 Update 1 now available from the Intel® Registration Center.

[Mariano_M_]

Hi again

I regret to communicate that the issue is NOT solved in Intel® Parallel Studio XE 2016 Update 1. 

The picture shows  that the sample program continues using the same size of memory as reported in my previous post, memory in use grows from 5.3 to 6.9 GB

 

Regards

[jimdempseyatthecove]

The memory shown in the performance monitor shows the memory for the Process that includes the pages of the heap that are free (and have been touched). Depending on the C Runtime Library Heap manager (Fortran uses this), a free to heap is or is not returned to the page pool free space (same with the Virtual Memory of the process). You can search msdn.microsoft.com for how you can select and tune the heap properties.

Jim Dempsey

[Mariano_M_]

Thanks Jim, but I still think that there is a bug in I/O, probably when closing files..

I've tried the following two programs (C and Fortran) both built with default settins from Visual Studio 2015; at the end of the loop the Fortran  program has 100 044 handles in use, while the  C program uses only 31.

If the loop is executed 200 000 times the number of handles used by the Fortran program is 200 044.

Regards

#include <windows.h>
#include <tchar.h>
#include <conio.h>

int main()
{
	HANDLE hFile;
	double a;
	int i;
	DWORD res;

	for(i = 0; i < 100000;i++)
	{
	hFile = CreateFile(_T("fil1.txt"), GENERIC_WRITE,0, 
		NULL,CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL, NULL); 
	a = i;
	WriteFile(hFile,&a, 8,&res,NULL); 
	CloseHandle(hFile);
	}
	_getch();
	return 0;
}

program Console2

    implicit none
    real*8 a
    integer i
    a=1.0
      do i=1,100000
        open(1,FILE='fil1.txt',status='REPLACE')
        a=i
     write (1,*)a
     close(1)
    enddo
    call sleepqq(10000)
    pause
    end program Console2

[IanH]

Under Tools > Options, in the Intel Compilers and Tools > Visual Fortran > Compilers category, make sure the selected compiler is "<Latest>".

I see no increase in handles with 16.0 update 1 with your test program in #14 in a vanilla VS2015 project.

[Mariano_M_]

Thanks Ian,

I've checked that setting and was "Latest" for both 32 and 64 architectures.

I've forced the selections to 2016.0.1.146 and the result is still the same: it increases one handle per instance of the loop.

Regards

Edit: I've uninstalled version 2016.0.0 and i get the same result.

[andrew_4619]

I assumed the handles bug was in the run time libraries,  is is possible you are using the older RTLs?

[Mariano_M_]

I've turn on the /verbose:LIB option and I've checked that  the fortran libraries used are all dated 10/25/2015, do you know if these are the latest available?

thanks

[Steven_L_Intel1]

Those are the latest. But maybe you have an old copy of libifcoremd.dll in PATH or in the executable folder.

Try this - change the "Use runtime library" property to "Multithreaded" instead of "Multithread DLL", rebuild, and see what you get.

[Mariano_M_]

Thanks all,

I've found old versions of some of the runtime libraries, that were not deleted when unistalling the 2016.0.0 version.

After replacing them with the latest the issue can be considered as SOLVED.

Regards