Can not predict a image using custom trained model.

[Sooyun]

Hello.

I want to predict a image using my custom trained model.

But, I got an error.

Please refer to following steps.

1. Set up the openvino environment.
- cd ~/usnoh/openvino_training_extensions/pytorch_toolkit/instance_segmentation
  source venv/bin/activate

2. Training from my images & annotations.
- rm -fr outputs/*
   export WORK_DIR=~/usnoh/rope_model
   export TRAIN_ANN_FILE="/home/ubuntu/usnoh/dataset/rope_dataset/instances_train.json"
   export TRAIN_IMG_ROOT="/home/ubuntu/usnoh/dataset/rope_dataset/train"
   export VAL_ANN_FILE="/home/ubuntu/usnoh/dataset/rope_dataset/instances_valid.json"
   export VAL_IMG_ROOT="/home/ubuntu/usnoh/dataset/rope_dataset/valid"
   python train.py \
   --batch-size 1 \
   --train-ann-files ${TRAIN_ANN_FILE} \
   --train-data-roots ${TRAIN_IMG_ROOT} \
   --val-ann-files ${VAL_ANN_FILE} \
   --val-data-roots ${VAL_IMG_ROOT} \
   --save-checkpoints-to ${WORK_DIR}/outputs

3. Converting trained model(.pth) to openvino xml.
- python export.py --load-weights outputs/latest.pth --save-model-to export --openvino

Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /home/ubuntu/usnoh/rope_model/export/model.onnx
- Path for generated IR: /home/ubuntu/usnoh/rope_model/export
- IR output name: model
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: boxes,labels,masks
- Input shapes: [1, 3, 384, 416]
- Mean values: [103.53, 116.28, 123.675]
- Scale values: [1.0, 1.0, 1.0]
- Scale factor: Not specified
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: False
ONNX specific parameters:
Model Optimizer version:

[ SUCCESS ] Generated IR version 10 model.
[ SUCCESS ] XML file: /home/ubuntu/usnoh/rope_model/export/model.xml
[ SUCCESS ] BIN file: /home/ubuntu/usnoh/rope_model/export/model.bin
[ SUCCESS ] Total execution time: 41.39 seconds.
[ SUCCESS ] Memory consumed: 624 MB.
It's been a while, check for a new version of Intel(R) Distribution of OpenVINO(TM) toolkit here https://software.intel.com/en-us/openvino-toolkit/choose-download?cid=&source=upgrade&content=2020_3_LTS or on the GitHub*

4. Prediction #1
  -  cd ~/omz_demos_build/intel64/Release
   ./mask_rcnn_demo -i ./111.bmp -m export/model.xml

InferenceEngine: API version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] ./111.bmp
[ INFO ] Loading Inference Engine
[ INFO ] Device info:
[ INFO ] CPU
MKLDNNPlugin version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2

[ INFO ] Loading network files
[ ERROR ] Cannot add output! Layer reshape_do_2d wasn't found!

5. Prediction #2

 -  ./mask_rcnn_demo -i ./111.bmp -m model.xml --detection_output_name masks
InferenceEngine: API version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] ./111.bmp
[ INFO ] Loading Inference Engine
[ INFO ] Device info:
[ INFO ] CPU
MKLDNNPlugin version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2

[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Network batch size is 1
[ INFO ] Prepare image ./111.bmp
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the device
[ INFO ] Create infer request
[ INFO ] Setting input data to the blobs
[ INFO ] Start inference
[ INFO ] Processing output blobs
[ ERROR ] AssertionFailed: do_blob->getTensorDesc().getDims().size() == 2

6. Prediction #3

./mask_rcnn_demo -i ./111.bmp -m model.xml -d GPU --detection_output_name masks
InferenceEngine: API version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] ./111.bmp
[ INFO ] Loading Inference Engine
[ INFO ] Device info:
[ INFO ] GPU
clDNNPlugin version ......... 2.1
Build ........... 2021.2.0-1877-176bdf51370-releases/2021/2

[ INFO ] Loading network files
[ INFO ] Preparing input blobs
[ INFO ] Network batch size is 1
[ INFO ] Prepare image ./111.bmp
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the device
[ ERROR ] Unknown Layer Type: ExperimentalDetectronROIFeatureExtractor

Please let me know if you have a solution.

Thanks.

[IntelSupport]

Hi Sooyun,

Thanks for reaching out.

For this error "[ ERROR ] Cannot add output! Layer reshape_do_2d wasn't found!". It seems like you need to input your layer specifically into your model. Also, you can run the Model Optimizer with --log_level DEBUG to see the inferred layers output shapes to see the mismatch. Refer to this documentation for more information.

https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_convert_model_tf_specific_Convert_Object_Detection_API_Models.html

"[ ERROR ] AssertionFailed: do_blob->getTensorDesc().getDims().size() == 2". Your model file might missing weights randomly. Please check your input shape at .xml file and make sure your input blob has the same input shape.

"[ ERROR ] Unknown Layer Type: ExperimentalDetectronROIFeatureExtractor" . The GPU plugin might not support the model. Can you try with the CPU plugin and see if the error arises or not.

Meanwhile, could you please share your model and the details of your system environment so that we can replicate this on our machine?

Regards,

Aznie

[Sooyun]

Hello Aznie.

Thanks for your reply.
I will refer to your answer.

And, you can download my dataset and model environment from below.
(Please reply when the download is complete. Then, I will disconnect the download link.)

* GPU : NVIDIA RTX6000 (NVIDIA-SMI 440.64, CUDA 10.2)
* Model training & export
- http://naver.me/FpXNrfUJ
* Dataset & coco annotations
- http://naver.me/xpPlktZb

I can provide all information to resolve this problem.
Please let me know if you need more information.

Thanks, Best regards.
Sooyun.

[Munesh_Intel]

Hi Sooyun,

I’ve downloaded both the folders. Thanks for sharing your model. I’ll investigate the issue and get back to you.

Please also note that Nvidia GPUs are not supported. OpenVINO requires Intel Processor Graphics GPU.

More information is available here: https://software.intel.com/content/www/us/en/develop/tools/openvino-toolkit/hardware.html

Regards,

Munesh

[Munesh_Intel]

Hi Soonhwan,

Thank you for waiting and apologies for the delay. We’ve tested your model with Benchmark App and verified that the model is working fine.

We’ve also tested your model with TensorFlow Object Detection Mask R-CNNs Segmentation C++ Demo. The explanation for your errors is given as follows:

For your Prediction number one: [ ERROR ] Cannot add output! Layer reshape_do_2d wasn't found!

This error is due to you are missing the ‘detection_output_name’ argument when running the demo. Thus, the demo will look for the default value "reshape_do_2d". This is mentioned in the following link:

https://docs.openvinotoolkit.org/2021.2/omz_demos_mask_rcnn_demo_README.html#running

For your Prediction number two: [ ERROR ] AssertionFailed: do_blob->getTensorDesc().getDims().size() == 2

For this error, it is due to differences/mismatches in input image info.

Refer to the links below:

https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/mask_rcnn_demo/main.cpp#L110

https://github.com/openvinotoolkit/open_model_zoo/blob/master/demos/mask_rcnn_demo/main.cpp#L222

Please also refer to the following links for more information:

InferenceEngine::InputInfo Class Reference

InferenceEngine::TensorDesc Class Reference

For your Prediction number three: [ ERROR ] Unknown Layer Type: ExperimentalDetectronROIFeatureExtractor

ExperimentalDetectronROIFeatureExtractor Layer is a legacy layer that has been deprecated and is no longer supported since the new concept of operation sets is introduced in OpenVINO 2020.1 version. 

https://docs.openvinotoolkit.org/2021.2/openvino_docs_MO_DG_prepare_model_convert_model_Legacy_IR_Layers_Catalog_Spec.html#ExperimentalDetectronROIFeatureExtractor

The probable reason for this error is that the layer is not supported by GPU.

We’ve also tested your model using Image Segmentation C++ Demo and Image Classification C++ Async Sample. The model is not suitable to be used for both these demo and sample as well as they support topologies with one output only.

Regards,

Munesh

[Sooyun]

Hi Munesh.

Sorry for late response and very thanks for your kindly answer.

First of all, I'm beginner of deep learning and openvino.

So, I can't fully understand your answer.

Regarding your answer, can I solve this problem if downgrade the openvino toolkit version from latest to 2019.R3.1?

I want to get prediction image with my custom trained rope model.

Which git tag of training extension should I use when learning model?

The training extension git repository does not have a version when I check.

Therefore, could you guide the openvino extension and toolkit properly version, git tag name?

I would very much appreciate if you could let me know in detail.

(Model train ~ predict a image demo with custom model)

Thanks, Best regards.
Sooyun

[Munesh_Intel]

Hi Sooyun,

Your model is compatible with the latest version of OpenVINO (2021.2), and there is no need for you to downgrade to previous versions of OpenVINO.

As we can see, the training extension link that you provide supports instance-segmentation-0904, instance-segmentation-0912, instance-segmentation-0228, instance-segmentation-0002 and instance-segmentation-0091 models. https://github.com/openvinotoolkit/training_extensions/blob/develop/pytorch_toolkit/instance_segmentation/model_templates/coco-instance-segmentation/readme.md#instance-segmentation

Step 1 (link given below) clearly mentions selecting one of the above models:

https://github.com/openvinotoolkit/training_extensions/blob/develop/pytorch_toolkit/instance_segmentation/model_templates/coco-instance-segmentation/readme.md#1-select-a-model-template-file-and-instantiate-it-in-some-directory

However, you seem to have retrained using a different model (Mask-RCNN: 'open-mmlab://detectron2/resnet50_caffe'), using a custom dataset.

Therefore, we would suggest you use one of the instance segmentation models mentioned above.

On another note, your present model did not work with TensorFlow Object Detection Mask R-CNNs Segmentation C++ Demo since it only supports image segmentation networks created with TensorFlow Object Detection API, as per

https://docs.openvinotoolkit.org/2021.2/omz_demos_mask_rcnn_demo_README.html

It potentially could work with Instance Segmentation Python Demo.

https://docs.openvinotoolkit.org/2021.2/omz_demos_python_demos_instance_segmentation_demo_README.html

Regards,

Munesh

[Munesh_Intel]

Hi Sooyun,

This thread will no longer be monitored since we have provided references, suggestions and solution. If you need any additional information from Intel, please submit a new question. 

Regards,

Munesh