valgrind memory leak with openvino inference

janshianliu · ‎07-15-2021

I am a beginner in openvino. Recently, I used valgrind to check the openvino sample programs. But the valgrind report shows that there is a memory leak in the program.

The sample program is a simple hello classification.

I used the official build_samples.sh to build all samples into executable files. And download the alexnet from Public Pre-Trained Models using downloader.py. Then valgrind the executable file of hello_classification. And I got the report as follow:

==5563== 9,760 (40 direct, 9,720 indirect) bytes in 1 blocks are definitely lost in loss record 357 of 357
==5563== at 0x4C33723: operator new(unsigned long) (vg_replace_malloc.c:417)
==5563== by 0x19C39559: ???
==5563== by 0x1ADA7CB1: ???
==5563== by 0x1ADACF16: ???
==5563== by 0x1A38D0EA: ???
==5563== by 0x19906E96: ???
==5563== by 0x1AE6E9A5: ???
==5563== by 0x9186906: __pthread_once_slow (pthread_once.c:116)
==5563== by 0x1ADC98FD: ???
==5563== by 0x1AC8CD7C: ???
==5563== by 0x1AC8E810: ???
==5563== by 0x19E5C460: ???
==5563== 
==5563== LEAK SUMMARY:
==5563== definitely lost: 6,696 bytes in 7 blocks
==5563== indirectly lost: 42,848 bytes in 448 blocks
==5563== possibly lost: 0 bytes in 0 blocks
==5563== still reachable: 17,231 bytes in 59 blocks
==5563== suppressed: 0 bytes in 0 blocks
==5563== Reachable blocks (those to which a pointer was found) are not shown.
==5563== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==5563== 
==5563== For lists of detected and suppressed errors, rerun with: -s
==5563== ERROR SUMMARY: 7 errors from 7 contexts (suppressed: 0 from 0)

It seems that a piece of memory has been new in a certain party without being deleted. Since I have no way to know exactly which file is causing it, I downloaded the source code from github and built it into debug mode.

I used valgrind again to test the same sample program. And I got the report as follow:

==3029== 64 bytes in 1 blocks are definitely lost in loss record 20 of 31
==3029== at 0x4C32FB5: malloc (vg_replace_malloc.c:380)
==3029== by 0x5450143: hwloc_linux_cpufreqs_init (in /opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==3029== by 0x5450754: look_sysfscpu (in /opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==3029== by 0x5453B3B: hwloc_linuxfs_look_cpu (in /opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==3029== by 0x5456CCC: hwloc_look_linuxfs (in /opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==3029== by 0x541BC74: hwloc_discover_by_phase (in /opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==3029== by 0x541BDCC: hwloc_discover (in /opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==3029== by 0x541D428: hwloc_topology_load (in /opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==3029== by 0x5411BB9: tbb::detail::r1::platform_topology::topology_initialization(unsigned long) (tbb_bind.cpp:84)
==3029== by 0x5412492: tbb::detail::r1::platform_topology::initialize(unsigned long) (tbb_bind.cpp:226)
==3029== by 0x541170A: __TBB_internal_initialize_system_topology (tbb_bind.cpp:446)
==3029== by 0x54055A3: custom::detail::initialize_system_topology()::{lambda()#1}::operator()() const (ie_parallel_custom_arena.cpp:83)
==3029== 
==3029== LEAK SUMMARY:
==3029== definitely lost: 128 bytes in 2 blocks
==3029== indirectly lost: 0 bytes in 0 blocks
==3029== possibly lost: 0 bytes in 0 blocks
==3029== still reachable: 3,606 bytes in 28 blocks
==3029== suppressed: 64 bytes in 1 blocks
==3029== Reachable blocks (those to which a pointer was found) are not shown.
==3029== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3029== 
==3029== For lists of detected and suppressed errors, rerun with: -s
==3029== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

After using the debug mode, I don’t know why pthread becomes tbb, but there is still a memory leak. And I still don’t know how to fix it.

Have anyone encountered a similar situation? How can I fix this problem?

Iffa_Intel · ‎07-15-2021

Hi,

Are you using the latest OpenVINO version 2021.4?

Please help to test this with the latest OpenVINO version.

Sincerely,

Iffa

janshianliu · ‎07-16-2021

No, I used the 2021.3 version in the previous report, and the report of the code I built later is version 2021.4.

Now, I downloaded the official version 2021.4 for testing. The valgrind report as follow:

==10432== 64 bytes in 1 blocks are definitely lost in loss record 20 of 31
==10432== at 0x4C32FB5: malloc (vg_replace_malloc.c:380)
==10432== by 0x521AF82: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x521B593: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x521E97A: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x5221B0B: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x51E6AB3: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x51E6C0B: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x51E8267: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x51DE901: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x923F906: __pthread_once_slow (pthread_once.c:116)
==10432== by 0x51DD1C7: ??? (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== by 0x516D48C: InferenceEngine::getAvailableCoresTypes() (in /opt/intel/openvino_2021.4.582/deployment_tools/inference_engine/lib/intel64/libinference_engine.so)
==10432== 
==10432== LEAK SUMMARY:
==10432== definitely lost: 128 bytes in 2 blocks
==10432== indirectly lost: 0 bytes in 0 blocks
==10432== possibly lost: 0 bytes in 0 blocks
==10432== still reachable: 3,606 bytes in 28 blocks
==10432== suppressed: 64 bytes in 1 blocks
==10432== Reachable blocks (those to which a pointer was found) are not shown.
==10432== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==10432== 
==10432== For lists of detected and suppressed errors, rerun with: -s
==10432== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

The content of the report is different from the 2021.3 version, but there is also a memory leak.

Iffa_Intel · ‎07-18-2021

we will further look into this and get back to you soon.

Sincerely,

Iffa

Andrey_S_Intel · ‎07-21-2021

It is a known issue https://github.com/open-mpi/hwloc/issues/382

Still exists in the latest OpenVINO 2021.4

Iffa_Intel · ‎09-06-2021

Hi,

We're currently trying to fix it and expect the fix to be available for OpenVINO 2021.4.2 release which is around the end of November.

Sincerely,

Iffa

Munesh_Intel · ‎12-09-2021

Hi JanShianLiu,

Thank you for your patience, and apologies for the late reply.

The memory leak issue has been fixed in the latest OpenVINO 2021.4.2. Please try from your end using OpenVINO 2021.4.2, and feel free to contact us should you face further issues.

Regards,

Munesh

Iffa_Intel · ‎12-16-2021

Greetings,

Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question.

Sincerely,

Iffa