I have been using Inspector XE to look for memory leaks and it is reporting a memory leak where I would not expect one from an MPI operation. The unusual behavior is that if the size of the transfer is below a certain array size no memory leaks are reported, but if the array size is larger than this threshold then a memory leak is reported.
I have attached a simple Fortran 90 program that demonstrates the behavior along with the makefile necessary to compile it.
The command to execute the program that I have been using is:
mpirun -np 2 inspxe-cl -r InspectorResults -collect mi3 <absolute path to executable>/test
If I set arraysize=10000 no memory leak is detected, but if I set the variable arraysize=20000 a memory leak is reported. Please see the screen shots included. both the send buffer and the receive buffer are reported as having memory leaks.
The versions of the Fortran compiler and Inspector XE I am using are:
[rick@RainierLinux testmpi]$ ifort -V
Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.0.0.084 Build 20101006
Copyright (C) 1985-2010 Intel Corporation. All rights reserved.
[rick@RainierLinux testmpi]$ inspxe-cl -V
Intel(R) Inspector XE 2011 Update 7 (build 189290) Command Line tool
Copyright (C) 2009-2011 Intel Corporation. All rights reserved.
Any thoughts on why a memory leak is being reported with one array size versus the other size?
I suspect the MPI implementation creates an internal buffer, and keeps it until the program ends. The two array sizes are about 39K and 78K (assume a 4-byte integer), which are suspiciously less than and greater than 64K (a nice, computer-sized number for a cutoff). Below the cutoff, either the MPI implementation does not use an internal buffer, or it has a pre-existing buffer of that size. Above the cutoff, it likely allocates a new buffer and keeps it for possible reuse (hence the reported leak).
If you double-click on the problem and go to the source view, there will probably be more stack frames extending into the MPI implementation (no source code with those frames). By default Inspector shows you the lowest location in the stack that has source code.
Your comments make sense as to the possible source for the memory leak reported by Inspector. Is there a way to configure the Intel MPI libraries to automatically use a larger internal buffer? If that is possible that would be an easy way to verify your hunch is the source of reported memory leak.