the result for faster_rcnn_resnet_101_coco model is wrong based on OpenVino in MYRIAD Plugin

[Hu__Can]

Hello, everyone 

when  I run the the python sample (object_detection_demo_ssd_async.py) on windows 10 with Intel® Neural Compute Stick 2 powered by the Intel® Movidius™ Myriad™ X ， I got following unbelievable and unreasonable result：

the code: res = exec_net.requests[cur_request_id].outputs[out_blob]

the result: [[[ [-1,0,0,0,0,0,0],  [0,0,0,0,0,0,0],  [0,0,0,0,0,0,0].....................

I download the pretrained model from here  

convert the model looks as follows:

C:\Program Files (x86)\IntelSWTools\openvino_2019.1.148\deployment_tools\model_optimizer>python mo_tf.py --input_model=frozen_inference_graph.pb --tensorflow_use_custom_operations_config faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --data_type FP16

the faster_rcnn_support.json is obtained from  C:\Program Files (x86)\IntelSWTools\openvino_2019.1.148\deployment_tools\model_optimizer\extensions\front\tf

the pipeline.config is along with frozen_inference_graph.pb in the downloaded file.

test model command line  : python object_detection_demo_ssd_async.py -i plane.jpg -m frozen_inference_graph.xml -d MYRIAD

However, the ssd_mobilenet_v1 is successful and result is good. is the the problem of converted  IR model ? or the faster_rcnn_resnet_101 model is currently not  supported ? But I got the message which said this model is supported, you  can see here

Besides, the process for plugin.load(network=net) is too long(at least one minute ) compared to ssd_mobilenet_v1 model.

Additionally，another strange thing is that  the input node in faster_rcnn_resnet_101 model contains two keys(image_tensor and image_info)  while the input node in ssd_mobilenet_v1 model only contains one key(image_tensor ),  is this a problem?

---------------------------------------------------

faster_rcnn_resnet_101 model info:
- the input node info 
   - 'image_tensor' # 1 3 300 300 NCHW FP32
   - 'image_info' # 1 3 NC FP32
- the output node info
   - 'detection_output' # 1 1 100 7 NCHW FP32

while the ssd_mobilenet_v1 model info:

- the input node info 
   - 'image_tensor' # 1 3 300 300 NCHW FP32
- the output node info
   - 'detection_output' # 1 1 100 7 NCHW FP32

------------------------------------------------

[Shubha_R_Intel]

Dear Hu, Can

In your default faster rcnn pipeline.config there is this part:

faster_rcnn {
    number_of_stages: 3
    num_classes: 90
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 800
        max_dimension: 1365
      }
    }

If the --input_shape command line parameter is not specified to your MO command (and I don't see --input_shape in your command above), the Model Optimizer generates an input layer with both height and width equal to the value of parameter min_dimension in the keep_aspect_ratio_resizer (in other words, 800x800,  a square image). If you're feeding in a non-square image and one that is not 800x800 your results could end up non-sensical  or generally bad. Kindly pay attention to the actual image size which Model Optimizer is assuming when you generate IR, you can see it if you use the --log_level DEBUG switch.

Is 800x800 in fact the correct image size that you are feeding into your model ? You say that you are using a pre-trained model (not custom-trained).

The Model Optimizer TF Object Detection API Doc explains it all in bits and pieces but I agree that it's very difficult to understand.

You can also reshape the topology using the IE API method "reshape" according to the rules specified in the pipeline.config file, but in this case use the "--keep_shape_ops" switch to Model Optimizer.

Please pay special attention to Important Notes About Feeding Input Images to the Samples .

With regard to this observation Besides, the process for plugin.load(network=net) is too long(at least one minute ) compared to ssd_mobilenet_v1 model.

Yes I wouldn't doubt it, though I haven't measured it myself. Long loading time is not an Inference Engine problem usually, it's due to the memory limitations of the Myriad VPU hardware.

With regard to this observation Additionally，another strange thing is that  the input node in faster_rcnn_resnet_101 model contains two keys(image_tensor and image_info)  while the input node in ssd_mobilenet_v1 model only contains one key(image_tensor ),  is this a problem?

Well faster_rcnn_resnet_101 and ssd_mobilenet_v1  are vastly different topologies so it's not surprising that their internal tensors and data structures don't look alike.

Hope I have answered your questions. Please report back here should you need further clarification.

Let me finalize by stating that when results are weird or aberrant such as the type you're seeing, it's almost always due to assumptions which Model Optimizer makes about pre-processing of input images, sometimes as simple as the assumptions it makes about input size. But keep in mind, Model Optimizer is just a graph compiler. It looks at trained models agnostically - it has no clue about how the model was trained, what pre-processing of images took place, etc...The human being has to tell model optimizer this information for it to produce the correct IR.

Hope this helps.

Thanks for using OpenVino !

Shubha

[Hu__Can]

Dear Shubha R:

thank you  for your reply .

According your suggestions,  I  used the new  command lines as follows like:

C:\Program Files (x86)\IntelSWTools\openvino_2019.1.148\deployment_tools\model_optimizer>python mo_tf.py --input_model=frozen_inference_graph.pb --tensorflow_use_custom_operations_config faster_rcnn_support.json --tensorflow_object_detection_api_pipeline_config pipeline.config --reverse_input_channels --data_type FP16 --input_shape [1,349,640,3]

The  size of tested  picture is 640*349(w*h), however, the result  is still the same as before. 

The model-tesing  command lines is as follow like:

python object_detection_demo_ssd_async.py -i plane.jpg -m frozen_inference_graph.xml -d MYRIAD

I don't  think  it's matter of this input_shape, even though  we set the incorrect input_shape, we cannot  get the unreasonable result ([-1，0，0，0，0，0，0], [-1，0，0，0，0，0，0].....................) in theory.   

is faster_rcnn_resnet_101 model not supported currently ?

The object_detection_demo_ssd_async.py  has been attached at the end of this reply with the type of .doc.

[Shubha_R_Intel]

Dear Hu, Can,

You are using MYRIAD X not NCS2.  The link you provided above (for where you obtained your model) doesn't work.  Can you instead obtain your model from The supported MO Tensorflow Object Detection API List ? And I see that Faster R-CNN ResNet 101 COCO is definitely supported.

Also can you try object_detection_demo instead of the SSD one ? Please do not use object_detection_demo_ssd_async.py . Unfortunately there is only a C++ version for object_detection_demo.

Please report your results here, and thanks for your patience,

Shubha