Yolo v3 model can not be processed by Model Optimizer

[Tsin__Ross]

Hi There,

while following steps described in 

https://software.intel.com/en-us/articles/OpenVINO-Using-TensorFlow

I got this 

(tensorflow) E:\AI\Data\OpenVINO>python tensorflow-yolo-v3/demo.py --weights_file yolov3-coco.weights --class_names coco.names --input_img pupils.jpg --output_img ./out.jpg
2019-01-10 00:43:11.216404: I d:\build\tensorflow\tensorflow-r1.9\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-01-10 00:43:11.222531: I d:\build\tensorflow\tensorflow-r1.9\tensorflow\core\common_runtime\process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.

(tensorflow) E:\AI\Data\OpenVINO>python C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\mo_tf.py --input_model  yolo_v3.pb --tensorflow_use_custom_operations_config  C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\extensions\front\tf\yolo_v3.json
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      E:\AI\Data\OpenVINO\yolo_v3.pb
        - Path for generated IR:        E:\AI\Data\OpenVINO\.
        - IR output name:       yolo_v3
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Offload unsupported operations:       False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\extensions\front\tf\yolo_v3.json
Model Optimizer version:        1.5.12.49d067a0
[ ERROR ]  Shape [ -1 416 416   3] is not fully defined for output 0 of "Placeholder". Use --input_shape with positive integers to override model input shapes.
[ ERROR ]  Cannot infer shapes or values for node "Placeholder".
[ ERROR ]  Not all output shapes were inferred or fully defined for node "Placeholder".
 For more information please refer to Model Optimizer FAQ (<INSTALL_DIR>/deployment_tools/documentation/docs/MO_FAQ.html), question #40.
[ ERROR ]
[ ERROR ]  It can happen due to bug in custom shape infer function <function tf_placeholder_ext.<locals>.<lambda> at 0x0000020515D7F400>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Stopped shape/value propagation at "Placeholder" node.
 For more information please refer to Model Optimizer FAQ (<INSTALL_DIR>/deployment_tools/documentation/docs/MO_FAQ.html), question #38.

How do I fix this?

Thank yo very much.

P.S. the demo.py under ./tensorflow-yolo-v3:

# -*- coding: utf-8 -*-

import numpy as np
import tensorflow as tf
from PIL import Image, ImageDraw
from tensorflow.python.framework import graph_io

from yolo_v3 import yolo_v3, load_weights, detections_boxes, non_max_suppression

FLAGS = tf.app.flags.FLAGS

tf.app.flags.DEFINE_string('input_img', '', 'Input image')
tf.app.flags.DEFINE_string('output_img', '', 'Output image')
tf.app.flags.DEFINE_string('class_names', 'coco.names', 'File with class names')
tf.app.flags.DEFINE_string('weights_file', 'yolov3.weights', 'Binary file with detector weights')

tf.app.flags.DEFINE_integer('size', 416, 'Image size')

tf.app.flags.DEFINE_float('conf_threshold', 0.5, 'Confidence threshold')
tf.app.flags.DEFINE_float('iou_threshold', 0.4, 'IoU threshold')


def load_coco_names(file_name):
    names = {}
    with open(file_name) as f:
        for id, name in enumerate(f):
            names[id] = name
    return names


def draw_boxes(boxes, img, cls_names, detection_size):
    draw = ImageDraw.Draw(img)

    for cls, bboxs in boxes.items():
        color = tuple(np.random.randint(0, 256, 3))
        for box, score in bboxs:
            box = convert_to_original_size(box, np.array(detection_size), np.array(img.size))
            draw.rectangle(box, outline=color)
            draw.text(box[:2], '{} {:.2f}%'.format(cls_names[cls], score * 100), fill=color)


def convert_to_original_size(box, size, original_size):
    ratio = original_size / size
    box = box.reshape(2, 2) * ratio
    return list(box.reshape(-1))


def main(argv=None):
    img = Image.open(FLAGS.input_img)
    img_resized = img.resize(size=(FLAGS.size, FLAGS.size))

    classes = load_coco_names(FLAGS.class_names)

    # placeholder for detector inputs
    inputs = tf.placeholder(tf.float32, [None, FLAGS.size, FLAGS.size, 3])

    with tf.variable_scope('detector'):
        detections = yolo_v3(inputs, len(classes), data_format='NHWC')
        load_ops = load_weights(tf.global_variables(scope='detector'), FLAGS.weights_file)

    boxes = detections_boxes(detections)
	
    with tf.Session() as sess:
        sess.run(load_ops)
		
        frozen = tf.graph_util.convert_variables_to_constants(sess, sess.graph_def, ['concat_1'])
        graph_io.write_graph(frozen, './', 'yolo_v3.pb', as_text=False)

        detected_boxes = sess.run(boxes, feed_dict={inputs: [np.array(img_resized, dtype=np.float32)]})

    filtered_boxes = non_max_suppression(detected_boxes, confidence_threshold=FLAGS.conf_threshold,
                                         iou_threshold=FLAGS.iou_threshold)

    draw_boxes(filtered_boxes, img, classes, (FLAGS.size, FLAGS.size))

    img.save(FLAGS.output_img)


if __name__ == '__main__':
    tf.app.run()

[Tsin__Ross]

I run again,

Different error:

(tensorflow) E:\AI\Data\OpenVINO>python tensorflow-yolo-v3/demo.py --weights_file yolov3-coco.weights --class_names coco.names --input_img pupils.jpg --output_img ./out.jpg
2019-01-10 01:16:30.906151: I d:\build\tensorflow\tensorflow-r1.9\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-01-10 01:16:30.914881: I d:\build\tensorflow\tensorflow-r1.9\tensorflow\core\common_runtime\process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.

(tensorflow) E:\AI\Data\OpenVINO>python C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\mo_tf.py --input_model  yolo_v3.pb --tensorflow_use_custom_operations_config C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\extensions\front\tf\yolo_v3.json
Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      E:\AI\Data\OpenVINO\yolo_v3.pb
        - Path for generated IR:        E:\AI\Data\OpenVINO\.
        - IR output name:       yolo_v3
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         Not specified, inherited from the model
        - Output layers:        Not specified, inherited from the model
        - Input shapes:         Not specified, inherited from the model
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       False
        - Reverse input channels:       False
TensorFlow specific parameters:
        - Input model in text protobuf format:  False
        - Offload unsupported operations:       False
        - Path to model dump for TensorBoard:   None
        - List of shared libraries with TensorFlow custom layers implementation:        None
        - Update the configuration file with input/output node names:   None
        - Use configuration file used to generate the model with Object Detection API:  None
        - Operations to offload:        None
        - Patterns to offload:  None
        - Use the config file:  C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\extensions\front\tf\yolo_v3.json
Model Optimizer version:        1.5.12.49d067a0
[ ERROR ]  Cannot infer shapes or values for node "detector/yolo-v3/meshgrid_1/mul_1/YoloRegion".
[ ERROR ]  index 2 is out of bounds for axis 0 with size 2
[ ERROR ]
[ ERROR ]  It can happen due to bug in custom shape infer function <function RegionYoloOp.regionyolo_infer at 0x000001DE82253EA0>.
[ ERROR ]  Or because the node inputs have incorrect values/shapes.
[ ERROR ]  Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ]  Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ]  Stopped shape/value propagation at "detector/yolo-v3/meshgrid_1/mul_1/YoloRegion" node.
 For more information please refer to Model Optimizer FAQ (<INSTALL_DIR>/deployment_tools/documentation/docs/MO_FAQ.html), question #38.

[Tsin__Ross · Accepted solution]

OK. worked with the following commands 

python tensorflow-yolo-v3-master/convert_weights_pb.py  --weights_file yolov3-coco.weights  --class_names coco.names  --size 416 --data_format NHWC

python C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\mo_tf.py --input_model frozen_darknet_yolov3_model.pb --tensorflow_use_custom_operations_config C:\Intel\computer_vision_sdk\deployment_tools\model_optimizer\extensions\front\tf\yolo_v3.json  --input_shape=[1,416,416,3] --data_type=FP32

[Koubele__Jakub]

Hi,you need to specify the input shape by adding the following argument: --input_shape [1,416,416,3]

[Nettleton__David]

Dear Sir/Madam,

Can you recommend the best yolo v3 source code to use to train our models so that they will work with openvino (with the runtime c++ code for the yolo v3 demo)?

Below are the details of our attempts.

Thankyou in advance and best regards,

David N.

----------------------------------------------------------

We wanted to train YOLO v3 model on our custom data set and feed it to model optimizer to be able to use openvino object_detection_demo_yolov3_async.py prediction script.

However, there are many implementations of the yolov3 available in github.

I don't know if the https://github.com/mystic123/tensorflow-yolo-v3 (mentioned in your website) is possible to train on my custom data.

Currently I am using yolov3 version of https://github.com/YunYang1994/tensorflow-yolov3 trained on my custom data.

However, when passed to the model optimizer it produced a lot of errors and with the help of openvino form I was able to solve some of those problems. Currently, the model optimizer is saying that some nodes are not converted and that is due to "FusedBatchNormalizationV3" usage, I know that in order to solve this problem I have to build custom layer.. However, even if I solve this the probability of making this implementation of yolov3 work with openvino is quite less and it would be really helpful if you can provide/suggest any source of yolov3 implementation that I can use to train on my custom data and also that is compatible with openvino.

----------------------------------------------------------

[Luis_at_Intel]

Hi Nettleton, David,

Thanks for reaching out. Please open a new thread so we can answer and help you there as this thread has already been closed. I'd be more than happy to assist you with any questions you may have.

Regards,

Luis