mkl 11.3.3 memory leak

[xian-zhong_guous_cd-]

We observed following memory leak from mkl:

10:57:09,158              7   3.179332e-08   7.099199e-08   2.239095e-14   8.614772e-15 
10:57:21,509 0 leaked Serializable objects
10:57:22,421 ==14604== 
10:57:22,427 ==14604== HEAP SUMMARY:
10:57:22,428 ==14604==     in use at exit: 1,009,739 bytes in 1,503 blocks
10:57:22,428 ==14604==   total heap usage: 2,080,855 allocs, 2,079,352 frees, 288,620,780 bytes allocated
10:57:22,428 ==14604== 
10:57:22,650 ==14604== 184 bytes in 1 blocks are possibly lost in loss record 608 of 730
10:57:22,653 ==14604==    at 0x4C26B5D: malloc (vg_replace_malloc.c:299)
10:57:22,654 ==14604==    by 0x24EE8AB6: mkl_serv_allocate (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_core.so)
10:57:22,654 ==14604==    by 0x22CBED34: DGELSY (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_lp64.so)
10:57:22,654 ==14604==    by 0x22EC02A0: LAPACKE_dgelsy_work (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_lp64.so)
10:57:22,654 ==14604==    by 0x22EC01C3: LAPACKE_dgelsy (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_lp64.so)

10:57:22,655 {
10:57:22,655    <insert_a_suppression_name_here>
10:57:22,655    Memcheck:Leak
10:57:22,655    match-leak-kinds: possible
10:57:22,655    fun:malloc
10:57:22,655    fun:mkl_serv_allocate
10:57:22,655    fun:DGELSY
10:57:22,656    fun:LAPACKE_dgelsy_work
10:57:22,656    fun:LAPACKE_dgelsy
10:57:22,656    fun:_Z10gradientReILi3EEvRSt6vectorI6VectorILi3EdESaIS2_EEjRKS0_IS1_IXT_EdESaIS6_EE
10:57:22,656    fun:_ZN10EvaluatorTI8BaryCellLi3EdE18prepareVertexBasedEv
10:57:22,656    fun:_ZN10EvaluatorTI8BaryCellLi3EdE9setOffsetERK7Element
10:57:22,656    fun:_ZNK13ViscousEngineI8BaryCelldE7reserveERK7ElementRK10FieldIndexI4CellER9AssemblerIdE
10:57:22,656    fun:_ZNK11DiscretizerIdE7exploreI4Cell8FvRegionEEvRKT0_RK9ModelPart
10:57:22,656    fun:_ZNK8TermEvalI20BasicRegionCondition22BasicConditionFaceTermE16SpecificDelegateclERK6FvTermRK8FvObject
10:57:22,656    fun:_ZN12FvTermHelper14evaluateOnPartI8FvRegionEEjRK6FvTermRKT_RK19TermAcceptorManagerb
10:57:22,656    fun:_ZN16VerifiedFaceTerm16verifiedEvaluateERK8FvDomainRKbS4_
10:57:22,656    fun:_ZN8FaceTerm8evaluateERK8FvDomain
10:57:22,656    fun:_ZN11DiscretizerIdE10discretizeERK8FvDomainb
10:57:22,656    fun:_ZN13ViscousSolver13solveSpecificIdEEvRK16FvRepresentationR14NamedResiduals
10:57:22,656    fun:_ZN13ViscousSolver15iterationUpdateER14NamedResiduals
10:57:22,656    fun:_ZN14RunnableSolver23doSolverIterationUpdateERK13SolverManagerR14NamedResiduals
10:57:22,656    fun:_ZN14RunnableSolver7iterateEv
10:57:22,656    fun:_ZN18SimulationIterator15startSimulationEP14RunnableSolveriNS_7RunModeEb
10:57:22,656    fun:_ZN14StepSimulation7executeERK10PropertiesRS0_
10:57:22,656    fun:_ZN10Controller14executeCommandER7CommandRK10PropertiesRS2_
10:57:22,657    fun:_ZN10Controller15processCommandsEv
10:57:22,657    fun:_ZN17CommandController23SerialMasterCommandLoop5startEv
10:57:22,657 }
10:57:22,657 ==14604== 256 bytes in 1 blocks are possibly lost in loss record 621 of 730
10:57:22,657 ==14604==    at 0x4C26B5D: malloc (vg_replace_malloc.c:299)
10:57:22,657 ==14604==    by 0x24EEA011: mm_account_ptr_by_tid..0 (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_core.so)
10:57:22,657 ==14604==    by 0x24EE83E9: mkl_serv_allocate (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_core.so)
10:57:22,657 ==14604==    by 0x22EC0176: LAPACKE_dgelsy (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_lp64.so)
10:57:22,657 ==14604==    by 0x2178E611: void gradientRe<3>(std::vector<Vector<3, double>, std::allocator<Vector<3, double> > >&, unsigned int, std::vector&lt;Vector<3, double>, std::allocator<Vector<3, double> > > const&) (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/star/lib/linux-x86_64-2.5/gnu4.8/lib/libViscousModel.so)


10:57:22,658 {
10:57:22,658    <insert_a_suppression_name_here>
10:57:22,658    Memcheck:Leak
10:57:22,658    match-leak-kinds: possible
10:57:22,659    fun:malloc
10:57:22,659    fun:mm_account_ptr_by_tid..0
10:57:22,659    fun:mkl_serv_allocate
10:57:22,659    fun:LAPACKE_dgelsy
10:57:22,659    fun:_Z10gradientReILi3EEvRSt6vectorI6VectorILi3EdESaIS2_EEjRKS0_IS1_IXT_EdESaIS6_EE
10:57:22,659    fun:_ZN10EvaluatorTI8BaryCellLi3EdE18prepareVertexBasedEv
10:57:22,659    fun:_ZN10EvaluatorTI8BaryCellLi3EdE9setOffsetERK7Element
10:57:22,659    fun:_ZNK13ViscousEngineI8BaryCelldE7reserveERK7ElementRK10FieldIndexI4CellER9AssemblerIdE
10:57:22,659    fun:_ZNK11DiscretizerIdE7exploreI4Cell8FvRegionEEvRKT0_RK9ModelPart
10:57:22,659    fun:_ZNK8TermEvalI20BasicRegionCondition22BasicConditionFaceTermE16SpecificDelegateclERK6FvTermRK8FvObject
10:57:22,659    fun:_ZN12FvTermHelper14evaluateOnPartI8FvRegionEEjRK6FvTermRKT_RK19TermAcceptorManagerb
10:57:22,659    fun:_ZN16VerifiedFaceTerm16verifiedEvaluateERK8FvDomainRKbS4_
10:57:22,659    fun:_ZN8FaceTerm8evaluateERK8FvDomain
10:57:22,659    fun:_ZN11DiscretizerIdE10discretizeERK8FvDomainb
10:57:22,659    fun:_ZN13ViscousSolver13solveSpecificIdEEvRK16FvRepresentationR14NamedResiduals
10:57:22,659    fun:_ZN13ViscousSolver15iterationUpdateER14NamedResiduals
10:57:22,659    fun:_ZN14RunnableSolver23doSolverIterationUpdateERK13SolverManagerR14NamedResiduals
10:57:22,659    fun:_ZN14RunnableSolver7iterateEv
10:57:22,659    fun:_ZN18SimulationIterator15startSimulationEP14RunnableSolveriNS_7RunModeEb
10:57:22,660    fun:_ZN14StepSimulation7executeERK10PropertiesRS0_
10:57:22,660    fun:_ZN10Controller14executeCommandER7CommandRK10PropertiesRS2_
10:57:22,660    fun:_ZN10Controller15processCommandsEv
10:57:22,660    fun:_ZN17CommandController23SerialMasterCommandLoop5startEv
10:57:22,660    fun:_ZN17CommandController15processCommandsEv
10:57:22,660 }
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ More log/stack-trace ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
10:57:22,666 ==14604== 557,216 bytes in 1 blocks are possibly lost in loss record 730 of 730
10:57:22,666 ==14604==    at 0x4C26B5D: malloc (vg_replace_malloc.c:299)
10:57:22,666 ==14604==    by 0x24EE8AB6: mkl_serv_allocate (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_core.so)
10:57:22,666 ==14604==    by 0x2EEF2954: mkl_blas_avx_xdtrsm (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_avx.so)
10:57:22,666 ==14604==    by 0x23739106: mkl_blas_dtrsm_omp_driver_v1 (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_thread.so)
10:57:22,666 ==14604==    by 0x237169BE: mkl_blas_dtrsm_host (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_thread.so)
10:57:22,666 ==14604==    by 0x237340D1: mkl_blas_dtrsm (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_thread.so)
10:57:22,666 ==14604==    by 0x2541ACC3: mkl_lapack_dgelsy (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_core.so)
10:57:22,666 ==14604==    by 0x22CBEDFA: DGELSY (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_lp64.so)
10:57:22,667 ==14604==    by 0x22EC02A0: LAPACKE_dgelsy_work (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_lp64.so)
10:57:22,667 ==14604==    by 0x22EC01C3: LAPACKE_dgelsy (in /home/install3/lin64/12.03.075_01/STAR-CCM+12.03.075/mkl/11.3.3/linux/lib/intel64/libmkl_intel_lp64.so)

10:57:22,668    <insert_a_suppression_name_here>
10:57:22,668    Memcheck:Leak
10:57:22,668    match-leak-kinds: possible
10:57:22,668    fun:malloc
10:57:22,668    fun:mkl_serv_allocate
10:57:22,668    fun:mkl_blas_avx_xdtrsm
10:57:22,668    fun:mkl_blas_dtrsm_omp_driver_v1
10:57:22,668    fun:mkl_blas_dtrsm_host
10:57:22,668    fun:mkl_blas_dtrsm
10:57:22,669    fun:mkl_lapack_dgelsy
10:57:22,669    fun:DGELSY
10:57:22,669    fun:LAPACKE_dgelsy_work
10:57:22,669    fun:LAPACKE_dgelsy
10:57:22,669    fun:_Z10gradientReILi3EEvRSt6vectorI6VectorILi3EdESaIS2_EEjRKS0_IS1_IXT_EdESaIS6_EE
10:57:22,669    fun:_ZN10EvaluatorTI8BaryCellLi3EdE18prepareVertexBasedEv
10:57:22,669    fun:_ZN10EvaluatorTI8BaryCellLi3EdE9setOffsetERK7Element
10:57:22,669    fun:_ZNK13ViscousEngineI8BaryCelldE7reserveERK7ElementRK10FieldIndexI4CellER9AssemblerIdE
10:57:22,669    fun:_ZNK11DiscretizerIdE7exploreI4Cell8FvRegionEEvRKT0_RK9ModelPart
10:57:22,669    fun:_ZNK8TermEvalI20BasicRegionCondition22BasicConditionFaceTermE16SpecificDelegateclERK6FvTermRK8FvObject
10:57:22,669    fun:_ZN12FvTermHelper14evaluateOnPartI8FvRegionEEjRK6FvTermRKT_RK19TermAcceptorManagerb
10:57:22,669    fun:_ZN16VerifiedFaceTerm16verifiedEvaluateERK8FvDomainRKbS4_
10:57:22,669    fun:_ZN8FaceTerm8evaluateERK8FvDomain
10:57:22,669    fun:_ZN11DiscretizerIdE10discretizeERK8FvDomainb
10:57:22,669    fun:_ZN13ViscousSolver13solveSpecificIdEEvRK16FvRepresentationR14NamedResiduals
10:57:22,669    fun:_ZN13ViscousSolver15iterationUpdateER14NamedResiduals
10:57:22,669    fun:_ZN14RunnableSolver23doSolverIterationUpdateERK13SolverManagerR14NamedResiduals
10:57:22,669    fun:_ZN14RunnableSolver7iterateEv
10:57:22,670 }
10:57:22,670 ==14604== LEAK SUMMARY:
10:57:22,670 ==14604==    definitely lost: 0 bytes in 0 blocks
10:57:22,670 ==14604==    indirectly lost: 0 bytes in 0 blocks
10:57:22,670 ==14604==      possibly lost: 627,584 bytes in 5 blocks
10:57:22,670 ==14604==    still reachable: 19,178 bytes in 150 blocks
10:57:22,670 ==14604==                       of which reachable via heuristic:
10:57:22,670 ==14604==                         stdstring          : 8,021 bytes in 130 blocks
10:57:22,670 ==14604==         suppressed: 362,977 bytes in 1,348 blocks
10:57:22,670 ==14604== Reachable blocks (those to which a pointer was found) are not shown.
10:57:22,670 ==14604== To see them, rerun with: --leak-check=full --show-leak-kinds=all
10:57:22,670 ==14604== 
10:57:22,670 ==14604== For counts of detected and suppressed errors, rerun with: -v
10:57:22,670 ==14604== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 3999 from 202)
10:57:22,741 Design STAR-CCM+ simulation completed
10:57:22,741 Server process exited with code : 231
10:57:22,745 

[Gennady_F_Intel]

Do you see mem leakage with dgelsy routine only?

have you try to set MKL_DISABLE_FAST environment variable to 1 or call mkl_disable_fast_mm() function?

[xian-zhong_guous_cd-]

We used mkl 11.1.1. before and start using mkl 11.3.3 recently. The memory leak seems only with mkl11.3.3. We will try the env variable MKL_DISABLE_FAST.

[xian-zhong_guous_cd-]

I got no memory leak using MKL 2017.2.174 with MKL_DISABLE_FAST=1. I am testing if MKL_DISABLE_FAST=1 is still necessary using MKL 2017.2.174. (With MKL11.3.3 and MKL_DISABLE_FAST=1, I still get some leaks.)

[xian-zhong_guous_cd-]

MKL_DISABLE_FAST=1 is not necessary if we use MKL 2017.2.174.