openvino error YoloV2 after it is transformed on Intermediate Representation

[rafael_kl]

I have trained a YOLOV2 using darkflow and now I want to run it in openvino. I have openvino R5 (2018.5.456) on windows 10

For this I transformed the weights into ".pb" and ."meta" and run the IR transformation ussing the documentation YOLOV1-V2 in https://software.intel.com/en-us/articles/OpenVINO-Using-TensorFlow

python3 mo_tf.py --input_model <path_to_model>/yolov2.pb --tensorflow_use_custom_operations_config <path_to_json>\yolo_v1_v2.json --batch 1

As my model has 2 classes I changed the original "yolo_v1_v2.json" file from <OPENVINO_INSTALL_DIR>/deployment_tools/model_optimizer/extensions/front/tf and put it in a <path_to_json> folder. My original model ("cfg" from darkflow) looks like this for the [regions] part

[region]
anchors =  0.57273, 0.677385, 1.87446, 2.06253, 3.33843, 5.47434, 7.88282, 3.52778, 9.77052, 9.16828
bias_match=1
classes=2
coords=4
num=5
softmax=1
jitter=.3
rescore=1

object_scale=5
noobject_scale=1
class_scale=1
coord_scale=1

absolute=1
thresh = .1
random=1

So I modified the yolo_v1_v2.json so it  looks like this for 2 classes:

[
  {
    "id": "TFYOLO",
    "match_kind": "general",
    "custom_attributes": {
      "classes": 2,
      "coords": 4,
      "num": 5,
      "do_softmax": 1
    }
  }
]

The Intermediate Representation was well created :

[ SUCCESS ] Generated IR model.
[ SUCCESS ] XML file: <path_to_lM>\yolov2.xml
[ SUCCESS ] BIN file: <path_to_IM>\yolov2.bin
[ SUCCESS ] Total execution time: 15.02 seconds.

Now when I try to use it to load the graph with the following python code:

plugin_dir = None
model_xml = <path_to_lM>\yolov2.xml
model_bin = <path_to_IM>\yolov2.bin

# Devices: GPU (intel), CPU, MYRIAD
plugin = IEPlugin("CPU", plugin_dirs=plugin_dir)

# Read IR
net = IENetwork.from_ir(model=model_xml, weights=model_bin)
assert len(net.inputs.keys()) == 1
assert len(net.outputs) == 1

input_blob = next(iter(net.inputs))
out_blob = next(iter(net.outputs))
# Load network to the plugin

exec_net = plugin.load(network=net)

I have an error :

exec_net = plugin.load(network=net)
  File "ie_api.pyx", line 389, in openvino.inference_engine.ie_api.IEPlugin.load
  File "ie_api.pyx", line 400, in openvino.inference_engine.ie_api.IEPlugin.load
RuntimeError: Unsupported primitive of type: ReorgYolo name: ExtractImagePatches

I know "ExtractImagePatches" is supported by openvino. The IM xml file looks like this:

<layer id="48" name="ExtractImagePatches" precision="FP32" type="ReorgYolo">
			<data stride="2"/>
			<input>
				<port id="0">
					<dim>1</dim>
					<dim>64</dim>
					<dim>38</dim>
					<dim>38</dim>
				</port>
			</input>
			<output>
				<port id="1">
					<dim>1</dim>
					<dim>256</dim>
					<dim>19</dim>
					<dim>19</dim>
				</port>
			</output>
		</layer>

I am new in openvino so maybe there is something I am missing. Help is really appreciated :) Thank you!

[Shubha_R_Intel]

Dear rafael, did you follow these instructions step-by-step ?

https://docs.openvinotoolkit.org/R5/_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_YOLO_From_Tensorflow.html

[rafael_kl]

Hello Shubha, yes I followed the documentation step by step and I tried adding to the code:

cpu_extension = "C:/Intel/computer_vision_sdk_2018.5.456/deployment_tools/inference_engine/src/extension/ext_reorg_yolo.cpp"

and it works well now. I could not find any documentation regarding this step (and I think the cpu_extention changed of folder compared to R4).

I need now help with another problem. I bought the NCS2 and I am trying to use opencv to execute the net (again I am working in windows, I dont know if it is supported). The code is the following:

import os
import sys
from openvino import inference_engine as ie
import cv2 as cv

model_xml = 'C:/Users/RK/Documents/Python/intel/IM/yolo.xml'
model_bin = 'C:/Users/RK/Documents/Python/intel/IM/yolo.bin'

# Load the model
net = cv.dnn.readNet(model_xml, model_bin)

 When I run this I get a message:

ModuleNotFoundError: No module named 'cv2'

I thought opencv was installed with the intel toolkit, I dont know how to acces it.

Anyway I installed opencv, I think version 4.0.0 (pip install opencv-python) and now I get this message:

net = cv.dnn.readNet(model_xml, model_bin)
cv2.error: OpenCV(4.0.1) D:\Build\OpenCV\opencv-4.0.1\modules\dnn\src\dnn.cpp:2633: error: (-2:Unspecified error) Build OpenCV with Inference Engine to enable loading models from Model Optimizer. in function 'cv::dnn::dnn4_v20181221::Net::readFromModelOptimizer'

I do not know what is going on, the path "D:\Build\OpenCV\opencv-4.0.1\modules\dnn\src\" dont even exist, how to use the good opencv to run my net in the NCS2?

Any leads could help a lot, thanks!

[Chakraborty__Subhasi]

I am getting my .xml file after converting a customized tiny-yolov2 model in Openvino R5 which is having its final output layer as follows :

<layer id="24" name="output/YoloRegion" precision="FP32" type="RegionYolo">
            <data axis="1" classes="8" coords="4" do_softmax="1" end_axis="3" num="5"/>
            <input>
                <port id="0">
                    <dim>1</dim>
                    <dim>65</dim>
                    <dim>13</dim>
                    <dim>13</dim>
                </port>
            </input>
            <output>
                <port id="1">
                    <dim>1</dim>
                    <dim>10985</dim>
                </port>
            </output>
        </layer>

But it is not matching in the required format as i am getting a vector as output but not a 3d matrix. What should i do?

[Shubha_R_Intel]

Subhasis, hopefully this thread will help you:

https://github.com/opencv/dldt/issues/88

Most notably, this part :

You can use this instruction and the custom operations configuration file with the following content to convert the yolo-v3 tiny:

[
{
"id": "TFYOLOV3",
"match_kind": "general",
"custom_attributes": {
"classes": 80,
"anchors": [10, 14, 23, 27, 37, 58, 81, 82, 135, 169, 344, 319],
"coords": 4,
"num": 6,
"mask": [0, 1, 2],
"entry_points": ["detector/yolo-v3-tiny/Reshape", "detector/yolo-v3-tiny/Reshape_4"]
}
}
]

[Shubha_R_Intel]

rafael, i have never had to install OpenCV individually. What version of OpenVino did you install ?  Did you run the setupvars.bat under bin first ?

[rafael_kl]

Shubha, thanks for your answer. I have the latest vino version R5 (2018.5.456) for Windows. Yes I set the enviroment variables using the setupvars batch file in bin. Normally I do this before running any openvino code. The log is the following:

C:\Users\Rafa>C:/intel/computer_vision_sdk_2018.5.456/bin/setupvars.bat
Python 3.6.5
Commande ECHO désactivée.
PYTHONPATH=C:\Intel\computer_vision_sdk_2018.5.456\python\python3.6;
[setupvars.bat] OpenVINO environment initialized

After that I try to run my code and I still get this error:

Traceback (most recent call last):
  File "test_rafa_2.py", line 4, in <module>
    import cv2 as cv
ModuleNotFoundError: No module named 'cv2'

I know the variables were initialissed because when I print

print(os.environ['PYTHONPATH'])
"C:\Intel\computer_vision_sdk_2018.5.456\python\python3.6;"

I dont know if it is normal. When I istalled openvino in my raspberry pi it worked perfectly and opencv 4.0.1 is well installed. I can run my models in the pi but I cannot code them in my computer since the py uses opencv to do inference and it doesnt work on my computer. My enviroment variables are attached.

Thanks for your help :)

[Monica__Zhao]

Hello, Chakraborty, Subhasis. I have the same situation as you. Have you solved it?