Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

MaskRCNN - C++ API - Reproduce Demo in Custom project

Ponchon__François
576 Views

Hey !

This post is linked with this Link Stackoverflow .


I would like to know what are exactly dependancies of MaskRCNN Demo ?

  • I used OpenCV DNN library in order to create an efficient inference engine "CustomIE".
  • I used successfully OpenVINO Model optimiser (python), to build the .xml and .bin file representing my network.
  • I successfully builded OpenVINO Sample directory with Visual Studio 2017 and run MaskRCNNDemo project with my network.

Finally, i wanted to change my OpenCV DNN "CustomIE" by an openvino based inference engine...

I reproduced almost exactly the same code than in Mask RCNN Demo, but i have different results....

With MaskRCNN_Demo :

mask_rcnn_demo.exe -m .\Release\frozen_inference_graph.xml -i .\Release\input.jpg

InferenceEngine:
        API version ............ 1.4
        Build .................. 19154
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ]     .\Release\input.jpg
[ INFO ] Loading plugin

        API version ............ 1.5
        Build .................. win_20181005
        Description ....... MKLDNNPlugin
[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ WARNING ] Image is resized from (4288, 2848) to (800, 800)
[ INFO ] Batch size is 1
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the plugin
[ INFO ] Start inference (1 iterations)

Average running time of one iteration: 2593.81 ms

[ INFO ] Processing output blobs
[ INFO ] Detected class 16 with probability 0.986519: [2043.3, 1104.9], [2412.87, 1436.52]
[ INFO ] Image out.png created!
[ INFO ] Execution successful

Bird_VinoCPP

Then everything work fine !

With my VinoBased IE :
 

Image is resized from (4288, 2848) to (800, 800)
Detected class 62 with probability 1: [4288, 0], [4288, 0]

 

I tried to watch "deeper" in order to understand what can be the issue....

I first had a look on dependancies :

<MaskRCNNDemo>
     //References
     <format_reader/>    => Open CV Images, resize it and get uchar data
     <ie_cpu_extension/> => CPU extension for un-managed layers (?)

     //Linker
     format_reader.lib         => Format Reader Lib (VINO Samples Compiled)
     cpu_extension.lib         => CPU extension Lib (VINO Samples Compiled)
     inference_engined.lib     => Inference Engine lib (VINO)
     opencv_world401d.lib      => OpenCV Lib
     libiomp5md.lib            => Dependancy
     ... (other libs)

On my own project :

<CustomIA>
     //References
     None => I use my own libtiff way to open image and i resize with OpenCV
     None => I will just add include to cpu_extension source code.

     //Linker
     opencv_world345d.lib   => OpenCV 3.4.5 library
     tiffd.lib              => Libtiff Library
     cpu_extension.lib      => CPU extension compiled with sample
     inference_engined.lib  => Inference engine lib.

+ Following dll

cpu_extension.dll
inference_engined.dll
libiomp5md.dll
mkl_tiny_omp.dll
MKLDNNPlugind.dll
opencv_world345d.dll
tiffd.dll
tiffxxd.dll

The cpu_extension.dll is the one compiled in the sample dir of OpenVino (idem for cpu_extension.lib.
Inference_engine.lib is the one from openvino directory.
 

I successfully compile but i face issue with plugin dispatcher :

Demo CODE:

 slog::info << "Loading plugin" << slog::endl;
    InferencePlugin plugin = PluginDispatcher({ FLAGS_pp, "../../../lib/intel64" , "" }).getPluginByDevice(FLAGS_d);

    /** Loading default extensions **/
    if (FLAGS_d.find("CPU") != std::string::npos) {
        /**
         * cpu_extensions library is compiled from "extension" folder containing
         * custom MKLDNNPlugin layer implementations. These layers are not supported
         * by mkldnn, but they can be useful for inferring custom topologies.
        **/
        plugin.AddExtension(std::make_shared<Extensions::Cpu::CpuExtensions>());
    }
    /** Printing plugin version **/
    printPluginVersion(plugin, std::cout);

OUTPUT :

[ INFO ] Loading plugin
    API version ............ 1.5
    Build .................. win_20181005
    Description ....... MKLDNNPlugin

My CODE:

   VINOEngine::VINOEngine()
{
    // Loading Plugin
    std::cout << std::endl;
    std::cout << "[INFO] - Loading VINO Plugin..." << std::endl;
    this->plugin= PluginDispatcher({ "", "../../../lib/intel64" , "" }).getPluginByDevice("CPU");
    this->plugin.AddExtension(std::make_shared<Extensions::Cpu::CpuExtensions>());
    printPluginVersion(this->plugin, std::cout);

OUTPUT :

[INFO] - Loading VINO Plugin...
000001A242280A18  // Like memory adress ???

Second issue is when i try to get boxes :

When i try to extract my ROI and masks from New Code, if i have a "match", i always have :

score =1.0

x1=x2=0.0

y1=y2=1.0

But the mask looks well extracted...

MYCODE :

 float score = box_info[2];
        if (score > this->Conf_Threshold)
        {
            // On reconstruit les coordonnées de la box..
            float x1 = std::min(std::max(0.0f, box_info[3] * Image.cols), static_cast<float>(Image.cols));
            float y1 = std::min(std::max(0.0f, box_info[4] * Image.rows), static_cast<float>(Image.rows));
            float x2 = std::min(std::max(0.0f, box_info[5] * Image.cols), static_cast<float>(Image.cols));
            float y2 = std::min(std::max(0.0f, box_info[6] * Image.rows), static_cast<float>(Image.rows));
            int box_width = std::min(static_cast<int>(std::max(0.0f, x2 - x1)), Image.cols);
            int box_height = std::min(static_cast<int>(std::max(0.0f, y2 - y1)), Image.rows);

Do anybody have an idea about what i make badly ?

How to create and link correctly an OpenVINO project using cpu_extension ?

Thanks !

0 Kudos
3 Replies
Shubha_R_Intel
Employee
576 Views

Francois, may I ask how you are running your executable ? I assume it's a command line  Windows app? Because in the command-line you do have to pass in a specific switch to add the full path cpu_extension.dll in order to actually execute the application. I see that you have statically linked cpu_extesion.lib with your application. Instead of doing that, can you try to load it through the command-line the way OpenVino samples do ? I think once you get your app working in this manner, you will be able to figure out why static linking doesn't work. There are probably some software pathways you are missing if you're not passing it in the DLL through the command-line. Then you can use something like Process Explorer to see if the DLL is actually being loaded.

-l "<absolute_path>" Optional. Absolute path to library with MKL-DNN (CPU) custom layers (*.so).

0 Kudos
Ponchon__François
576 Views

Shubha R. (Intel) wrote:

Francois, may I ask how you are running your executable ? I assume it's a command line  Windows app? Because in the command-line you do have to pass in a specific switch to add the full path cpu_extension.dll in order to actually execute the application. I see that you have statically linked cpu_extesion.lib with your application. Instead of doing that, can you try to load it through the command-line the way OpenVino samples do ? I think once you get your app working in this manner, you will be able to figure out why static linking doesn't work. There are probably some software pathways you are missing if you're not passing it in the DLL through the command-line. Then you can use something like Process Explorer to see if the DLL is actually being loaded.

-l "<absolute_path>" Optional. Absolute path to library with MKL-DNN (CPU) custom layers (*.so).



Hello, thanks for your answer !
I run my program directly from the degugger like that :

CustomIE.exe "PathtoImage.jpg"

The program contains a "main.cpp" like that :

int main(int argc,char** argv)
{
	std::string FilePath;
	VINOEngine MaskVino =VINOEngine();
	//CNNEngine MaskCV = CNNEngine();
	FilePath = argv[1];
	bool isDisplayed = true;
	if (argc > 2)
	{
		if (strcmp(argv[2],"-NoDisplay")==0)
		{
			isDisplayed = false;
		}
	}
	//MaskCV.ProcessImage(FilePath,1);
	MaskVino.ProcessImage("input.jpg", true,false);

	return 0;
}

And i build the vino engine with the constructor like i described in the post just before.

How to pass the path of cpu_extension.dll ? Because i already call a "plugindispatcher" with path set to "./" containing a compiled cpu_extension.dll.


The problem is that i don't have any exception, it seems to work like a charm except that detection is bad and i don't seem to load plugin correctly unfortunately.

Unfortunately, in my app, i don't manage the -l args, because it is not based exactly on the "example".
I watched the code of mask_rcnn_sample, and if "-l" args is set, it seems to wait a ".so" file. But as you noticed, i'am on windows, then i can't manage such files this way.

I'll try to inspect with process Explorer.

Hope you'll find a way to help me :).

if it can help, i ll show you the way i linked my program :

Include

VinoInclude.JPG

Linker

VinoLinker1.JPG

VinoLinker2.JPG

0 Kudos
Ponchon__François
576 Views

I tried to use "process explorer", especially "ListDlls" Tool.
I had the following result :
 

Command line: "C:\Users\fponc\Documents\Visual Studio 2017\Projects\CustomIE\x64\Debug\CustomIE.exe" D:\\Dev\\Python\\13-TensorFlow_ObjectDetection\\05899706_k_p1.tif

Base                Size      Path
0x000000001bd40000  0x1b3000  <path>\Debug\CustomIE.exe
0x0000000062790000  0x1e1000  C:\WINDOWS\SYSTEM32\ntdll.dll
0x00000000420a0000  0x12000   C:\Program Files\AVAST Software\Avast\aswhook.dll
0x00000000602b0000  0xb2000   C:\WINDOWS\System32\KERNEL32.DLL
0x000000005f800000  0x273000  C:\WINDOWS\System32\KERNELBASE.dll
0x00000000036b0000  0x73c6000  <path>\Debug\opencv_world345d.dll
0x0000000062590000  0x28000   C:\WINDOWS\System32\GDI32.dll
0x000000005f5b0000  0x192000  C:\WINDOWS\System32\gdi32full.dll
0x000000005fa80000  0x9f000   C:\WINDOWS\System32\msvcp_win.dll
0x000000005fb20000  0xfa000   C:\WINDOWS\System32\ucrtbase.dll
0x00000000625c0000  0x190000  C:\WINDOWS\System32\USER32.dll
0x000000005eb60000  0x20000   C:\WINDOWS\System32\win32u.dll
0x0000000061fb0000  0x151000  C:\WINDOWS\System32\ole32.dll
0x000000005fd20000  0x322000  C:\WINDOWS\System32\combase.dll
0x0000000060170000  0x124000  C:\WINDOWS\System32\RPCRT4.dll
0x000000005eb80000  0x7a000   C:\WINDOWS\System32\bcryptPrimitives.dll
0x00000000604f0000  0x5b000   C:\WINDOWS\System32\sechost.dll
0x0000000061ee0000  0xc2000   C:\WINDOWS\System32\OLEAUT32.dll
0x0000000062110000  0xed000   C:\WINDOWS\System32\COMDLG32.dll
0x0000000062200000  0x9e000   C:\WINDOWS\System32\msvcrt.dll
0x0000000060060000  0xa9000   C:\WINDOWS\System32\shcore.dll
0x000000005fcc0000  0x51000   C:\WINDOWS\System32\SHLWAPI.dll
0x0000000060550000  0x1440000  C:\WINDOWS\System32\SHELL32.dll
0x000000005f560000  0x49000   C:\WINDOWS\System32\cfgmgr32.dll
0x000000005ee50000  0x70d000  C:\WINDOWS\System32\windows.storage.dll
0x00000000622a0000  0xa1000   C:\WINDOWS\System32\advapi32.dll
0x000000005eb10000  0x11000   C:\WINDOWS\System32\kernel.appcore.dll
0x000000005eb40000  0x1f000   C:\WINDOWS\System32\profapi.dll
0x000000005eac0000  0x4c000   C:\WINDOWS\System32\powrprof.dll
0x000000005eb30000  0xa000    C:\WINDOWS\System32\FLTLIB.DLL
0x0000000030c40000  0xca000   <path>\Debug\tiffd.dll
0x00000000235e0000  0x18c000  <path>\Debug\cpu_extension.dll
0x00000000124c0000  0x17cf000  <path>\Debug\inference_engined.dll
0x0000000022ca0000  0xf1000   C:\WINDOWS\SYSTEM32\MSVCP140D.dll
0x000000004a7b0000  0x22000   C:\WINDOWS\SYSTEM32\VCRUNTIME140D.dll
0x0000000022570000  0x1b8000  C:\WINDOWS\SYSTEM32\ucrtbased.dll
0x000000005aeb0000  0xa7000   C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_5.82.17134.523_none_f950727330975f7b\COMCTL32.dll
0x000000002eb50000  0xba000   C:\WINDOWS\SYSTEM32\CONCRT140D.dll
0x00000000583f0000  0x1db000  C:\WINDOWS\SYSTEM32\MFPlat.DLL
0x00000000510f0000  0x7b000   C:\WINDOWS\SYSTEM32\MF.dll
0x000000002a2f0000  0xf8000   C:\WINDOWS\SYSTEM32\MFReadWrite.dll
0x000000005bf90000  0x30b000  C:\WINDOWS\SYSTEM32\d3d11.dll
0x00000000223a0000  0x1c9000  <path>\Debug\libiomp5md.dll
0x00000000203e0000  0x378000  C:\Program Files\Common Files\Intel\Shared Libraries\redist\intel64_win\compiler\libmmdd.dll
0x000000005d860000  0xbb000   C:\WINDOWS\SYSTEM32\dxgi.dll
0x0000000052220000  0x42e000  C:\WINDOWS\SYSTEM32\MFCORE.DLL
0x000000005ec60000  0x1e2000  C:\WINDOWS\System32\CRYPT32.dll

The interesting information is that cpu_extension are correctly loaded :S

0 Kudos
Reply