Both 2019 impi releases have a severe memory leak which goes away when I regress to the 2015 version (i.e. source /opt/intel/comp2015/impi/5.0.2.044/intel64/bin/mpivars.sh). I am attaching two valgrind outputs, lapw1.vg.285276 from 2019 impi and lapw1.vg.5451 from 2015 impi which show it quite clearly.
For reference, the entries in the valgrind logs with "init_parallel_ (in /opt/Wien2k_18.1F/lapw1Q_mpi)" are the mpi initialization, so these are almost certainly not real as the initialization is only done once. The entries associated with the scalapack pdsygst call are probably the culprit.
If needed I can provide a package to reproduce this. It is part of a large code, so decomposing into a small test code is not feasible.
Can I ask for this issue to be escalated. I reported it to Intel on 11/26/2018 and it is request 03837257. Beyond confirmation that it is probably in impi (which I already had verified) it does not appear to be going anywhere.
I strongly suspect that this bug is a very significant roadblock in the use of impi for the Wien2k software package (www.wien2k.at) which has about 12000 citations on Google scholar and has been licenced to more than 3000 users groups around the world. So far I have myself documented it on my cluster, one in Romania and more recently when testing a cluster from a vendor in the US.
On 04/03/2019 the engineering team response to 03837257 (the ticket related to this) indicated that it might be solved with Upgrade 4. They have not confirmed with me that it is.